Nov 02 14:55:56.215071 ip-10-0-135-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:55:56.611478 ip-10-0-135-103 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:55:56.611478 ip-10-0-135-103 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:55:56.611478 ip-10-0-135-103 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:55:56.611478 ip-10-0-135-103 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:55:56.615587 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615508 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.615587 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615586 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615590 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615594 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615598 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615601 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615604 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615607 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615610 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615613 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615616 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615619 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615622 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615625 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615628 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615631 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615634 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615637 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615640 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615643 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.615652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615646 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615649 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615652 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615655 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615658 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615665 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615668 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615671 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615673 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615676 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615678 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615681 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615684 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615686 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615689 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615691 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615694 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615696 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615699 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615701 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.616105 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615704 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615707 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615709 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615712 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615714 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615717 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615719 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615722 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615724 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615727 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615729 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615732 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615734 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615738 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615741 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615744 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615747 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615750 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615752 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615755 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.616576 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615758 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615760 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615765 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615772 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615776 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615780 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615783 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615786 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615789 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615792 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615812 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615819 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615826 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615830 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615833 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615835 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.617047 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615838 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615841 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615844 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615849 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615852 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615855 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615858 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615861 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615864 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615867 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615869 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615872 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615875 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615878 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615880 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615883 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615886 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615888 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.617490 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615891 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615894 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615896 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615899 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615902 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615907 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615911 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.615915 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616696 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616705 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616708 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616711 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616713 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616718 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616722 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616725 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616728 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616731 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616734 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.617952 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616737 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616740 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616743 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616746 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616749 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616751 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616754 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616757 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616760 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616763 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616766 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616768 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616771 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616774 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616776 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616782 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616785 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616787 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616790 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.618410 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616793 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616809 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616812 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616818 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616821 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616823 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616826 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616828 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616831 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616833 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616836 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616843 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616846 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616848 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616851 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616854 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616856 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.618894 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616859 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616862 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616868 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616870 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616873 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616875 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616880 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616883 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616885 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616889 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616892 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616896 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616900 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616903 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616906 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616909 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616912 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616914 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.619356 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616918 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616921 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616923 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616926 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616929 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616931 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616934 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616936 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616939 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616942 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616945 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616948 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616951 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616954 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616956 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616959 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616962 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616964 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616967 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616970 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.619830 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616973 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616976 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616979 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616981 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616984 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616987 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616989 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616992 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616995 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.616999 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617001 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617004 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617006 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617009 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617014 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617503 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617509 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617511 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.620307 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617514 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617517 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617520 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617522 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617525 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617528 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617531 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617534 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617537 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617540 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617543 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617545 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617549 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617551 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617554 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617557 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617560 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617562 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617565 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.620776 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617568 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617570 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617574 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617576 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617579 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617582 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617584 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617587 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617589 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617592 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617595 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617597 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617600 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617602 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617605 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617608 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617610 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617613 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.621253 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617618 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617621 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617624 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617627 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617630 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617633 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617636 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617641 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617645 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617649 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617651 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617654 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617657 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617659 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617662 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617666 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617669 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617672 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.621688 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617675 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617677 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617680 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617684 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617688 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617691 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617694 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617696 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617699 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617701 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617704 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617707 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617710 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617712 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617715 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617721 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617723 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617726 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617729 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617731 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.622189 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617734 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617737 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617739 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617742 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617745 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617747 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617750 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617753 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617755 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617758 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617761 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617764 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617766 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617769 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617772 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617775 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617777 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617780 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617782 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.622652 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617787 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617790 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617793 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617808 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617811 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617814 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.617817 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618720 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618734 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618745 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618751 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618756 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618759 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618767 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618771 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618775 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618778 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618781 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618785 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618788 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618791 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618795 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618809 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:55:56.623145 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618813 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618817 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618821 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618824 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618827 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618830 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618834 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618839 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618842 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618847 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618850 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618853 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618857 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618860 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618863 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618866 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618871 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618875 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618878 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618882 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618885 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618888 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618893 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618896 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 14:55:56.623760 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618899 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618902 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618906 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618910 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618913 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618916 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618919 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618923 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618926 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618929 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618932 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618935 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618938 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618941 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618946 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618949 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618952 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618956 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618959 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618962 2579 flags.go:64] FLAG: --help="false" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618965 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618969 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618972 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:55:56.624379 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618975 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618979 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618983 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618986 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618989 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618992 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618995 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.618998 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619001 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619004 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619007 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619009 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619013 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619016 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619019 2579 flags.go:64] FLAG: --lock-file="" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619021 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619024 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619028 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619033 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619036 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619040 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619043 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619046 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619050 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:55:56.624954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619052 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619055 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619060 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619063 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619067 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619070 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619074 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619077 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619080 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619083 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619086 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619089 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619096 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619099 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619103 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619105 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619108 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619111 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619114 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619117 2579 flags.go:64] FLAG: --port="10250" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619120 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619123 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01b0ea0a7f519b1e0" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619127 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619130 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619133 2579 flags.go:64] FLAG: --register-node="true" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619136 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:55:56.625507 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619139 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619142 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619145 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619148 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619152 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619155 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619158 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619161 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619164 2579 flags.go:64] FLAG: --runonce="false" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619168 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619171 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619174 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619177 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619180 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619183 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619186 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619190 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619193 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619196 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619199 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619202 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619205 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619208 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619211 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619216 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:55:56.626119 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619219 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619222 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619226 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619229 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619232 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619235 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619238 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619241 2579 flags.go:64] FLAG: --v="2" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619245 2579 flags.go:64] FLAG: --version="false" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619249 2579 flags.go:64] FLAG: --vmodule="" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619253 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619257 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619395 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619399 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619403 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619406 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619409 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619411 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619414 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619417 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619420 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619422 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619425 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.626742 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619428 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619431 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619433 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619436 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619439 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619442 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619445 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619449 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619453 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619455 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619458 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619461 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619463 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619466 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619469 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619471 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619474 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619477 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619480 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.627297 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619482 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619485 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619488 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619491 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619498 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619501 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619503 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619506 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619509 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619511 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619514 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619518 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619521 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619523 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619526 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619529 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619531 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619534 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619537 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.627747 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619539 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619542 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619545 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619547 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619550 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619553 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619555 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619558 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619560 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619563 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619565 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619569 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619571 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619574 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619576 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619580 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619582 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619589 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619592 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.628235 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619594 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619597 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619600 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619602 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619605 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619609 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619612 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619614 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619617 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619620 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619622 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619625 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619628 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619631 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619633 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619636 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619639 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619641 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619644 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.628699 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619647 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619649 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619654 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619657 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619660 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619663 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619665 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619668 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619670 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619673 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619678 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619680 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619683 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619686 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619689 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619693 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.629251 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619697 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.619727 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619762 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619767 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619770 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619773 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619776 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619780 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619784 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619787 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619791 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619794 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619808 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619811 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619814 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.629669 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619816 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619819 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619822 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619825 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619828 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619831 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619837 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619840 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619843 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619845 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619850 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619853 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619856 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619859 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619861 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619864 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619868 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619872 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619876 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.630048 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619880 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619883 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619885 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619888 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619891 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619894 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619896 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619899 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619902 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619905 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619907 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619910 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619913 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619916 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619918 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619921 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619925 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619929 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619932 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619934 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.630515 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619937 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619940 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619942 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619946 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619949 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619952 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619955 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619958 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619960 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619963 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619966 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619970 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619972 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619975 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619978 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619981 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619983 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619986 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619989 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619991 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.630989 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619994 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.619997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620000 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620002 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620005 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620008 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620010 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620013 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620016 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620018 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620021 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620024 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620026 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620029 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620032 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620035 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620038 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620041 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620044 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620047 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.631460 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620049 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620052 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620055 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620059 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620061 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620064 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620067 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620069 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620072 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620076 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620079 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620081 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.620084 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.627275 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.627297 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.627997 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.628004 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628062 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628068 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.631934 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628071 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628077 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628080 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628083 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628086 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628089 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628091 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628094 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628097 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628099 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628104 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628108 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628111 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628114 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628117 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628120 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628123 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628125 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.632415 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628128 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628131 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628134 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628137 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628140 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628142 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628145 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628147 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628151 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628154 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628157 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628160 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628163 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628166 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628169 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628172 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628174 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628177 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628180 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.632937 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628182 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628185 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628188 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628190 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628193 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628196 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628198 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628201 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628204 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628207 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628209 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628212 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628215 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628218 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628221 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628224 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628226 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628229 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628232 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.633381 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628239 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628241 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628246 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628250 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628257 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628260 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628264 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628266 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628269 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628272 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628275 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628277 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628280 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628283 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628285 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628288 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628291 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628294 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628296 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.633855 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628299 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628302 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628304 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628307 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628310 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628313 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628316 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628319 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628321 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628325 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628327 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628330 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628333 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628336 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628338 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628348 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628352 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628356 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628358 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628361 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.634345 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628364 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628367 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628369 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628372 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628375 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628378 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.628406 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628549 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628553 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628557 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628560 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628563 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628566 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628569 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628572 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.634824 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628575 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628578 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628580 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628583 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628586 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628588 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628592 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628597 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628600 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628602 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628605 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628608 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628611 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628614 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628617 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628619 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628622 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628625 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628627 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.635174 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628630 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628633 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628635 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628638 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628641 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628643 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628646 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628649 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628652 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628654 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628657 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628659 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628662 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628665 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628667 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628670 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628673 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628675 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628678 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.635646 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628680 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628683 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628686 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628689 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628692 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628694 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628697 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628699 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628703 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628705 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628708 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628711 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628714 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628716 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628719 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628721 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628724 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628727 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628729 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.636167 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628732 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628734 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628737 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628740 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628742 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628746 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628749 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628753 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628756 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628759 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628762 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628765 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628767 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628770 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628773 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628775 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628778 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628780 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628783 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.636854 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628786 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628788 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628791 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628794 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628810 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628815 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628818 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628822 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628827 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628830 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628833 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628838 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628841 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628843 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628846 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628849 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628851 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628854 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628857 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.637300 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628860 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628862 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.628890 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628921 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628925 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628928 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628931 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628934 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628937 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628939 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628942 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628944 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628947 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628949 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628952 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628955 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:55:56.637841 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628958 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628961 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628966 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628969 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628972 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628975 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628977 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628980 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628983 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628986 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628988 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628991 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628993 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628996 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.628999 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629001 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629004 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629008 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:55:56.638216 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629010 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629013 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629015 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629018 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629020 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629023 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629026 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629028 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629031 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629033 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629036 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629041 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629044 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629047 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629050 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629053 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629055 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629058 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629060 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:55:56.638906 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629063 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629065 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629069 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629073 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629077 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629080 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629082 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629085 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629087 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629090 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629093 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629096 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629099 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629102 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629104 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629107 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629109 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629112 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629114 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:55:56.639565 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629117 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629120 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629123 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629126 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629129 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629131 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629134 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629138 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629141 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629144 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629146 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629149 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629152 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629155 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629157 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629160 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629163 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629166 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629169 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629171 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:55:56.640065 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629174 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629176 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629179 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629182 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629185 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629188 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629191 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629194 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629196 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629199 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629201 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629204 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629207 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629212 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:56.629216 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.629267 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.631182 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.632086 2579 server.go:1014] "Starting client certificate rotation" Nov 02 14:55:56.640598 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.632185 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:55:56.641058 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.632218 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:55:56.641058 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.634165 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:55:56.641058 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.639407 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:55:56.648615 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.648598 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:55:56.651714 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.651698 2579 log.go:25] "Validated CRI v1 image API" Nov 02 14:55:56.652554 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.652526 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:55:56.656636 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.656619 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a28a6853-da96-4013-8309-32ad9e1f2f33:/dev/nvme0n1p3 c7983892-a0a0-49a4-a892-db41c1bb7529:/dev/nvme0n1p4] Nov 02 14:55:56.656690 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.656636 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 14:55:56.661643 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.661536 2579 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:56.660259057 +0000 UTC m=+0.329546146 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099089 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27cc97ddbb408310eef35eb35480d7 SystemUUID:ec27cc97-ddbb-4083-10ee-f35eb35480d7 BootID:08b01d73-5f3b-43d3-a941-8889b7ebca23 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:27:d4:85:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:27:d4:85:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:b4:a1:b9:43:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:55:56.662417 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.662406 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:55:56.662492 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.662480 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:55:56.664861 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.664842 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:55:56.664981 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.664862 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-103.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:55:56.665027 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.664982 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:55:56.665027 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.664990 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:55:56.665027 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.665025 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:55:56.665103 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.665042 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:55:56.665862 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.665853 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:55:56.669548 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.669537 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:55:56.669688 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.669679 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:55:56.670428 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670420 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:55:56.670459 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670432 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:55:56.670459 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670443 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:55:56.670459 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670454 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:55:56.670541 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670462 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:55:56.670739 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.670724 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:55:56.673392 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.673376 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:55:56.678962 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.678751 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:55:56.680604 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680586 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:55:56.680656 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680622 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:55:56.680950 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680935 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:55:56.680983 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680957 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:55:56.680983 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680964 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:55:56.680983 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680969 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:55:56.680983 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680975 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:55:56.680983 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.680982 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:55:56.681512 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681495 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:55:56.681565 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681520 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:55:56.681565 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681529 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:55:56.681565 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681536 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:55:56.681565 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681547 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:55:56.681565 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.681557 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:55:56.682151 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.682140 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:55:56.682151 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.682151 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:55:56.685688 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.685675 2579 server.go:1263] "Started kubelet" Nov 02 14:55:56.685812 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.685775 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:55:56.685851 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.685767 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:55:56.685851 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.685840 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:55:56.686395 ip-10-0-135-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:55:56.686929 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.686906 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:55:56.689415 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.689402 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:55:56.691890 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.691868 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 14:55:56.692224 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.692205 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:55:56.692661 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.692644 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:55:56.692757 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.692740 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693240 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693435 2579 factory.go:55] Registering systemd factory Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693452 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693652 2579 factory.go:153] Registering CRI-O factory Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693667 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 14:55:56.693719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693719 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693738 2579 factory.go:103] Registering Raw factory Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693754 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693829 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693847 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693979 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:55:56.694141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.693985 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:55:56.694637 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.694610 2579 manager.go:319] Starting recovery of all containers Nov 02 14:55:56.704094 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.704067 2579 manager.go:324] Recovery completed Nov 02 14:55:56.709454 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.709440 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.711261 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.711247 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.711326 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.711275 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.711326 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.711288 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.712846 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.712673 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:55:56.712846 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.712686 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.712846 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.712687 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:55:56.712846 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.712726 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:55:56.713400 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.713380 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.713486 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.713411 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.713486 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.713421 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.714949 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.714932 2579 policy_none.go:50] "Start" Nov 02 14:55:56.715017 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.714956 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:55:56.715017 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.714971 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:55:56.716997 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.716986 2579 policy_none.go:44] "Start" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.743545 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.751557 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.751593 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.751602 2579 server.go:87] "Starting device plugin registration server" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.751902 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.751913 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.752014 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.752099 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.752108 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.752743 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.752784 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.757054 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.758180 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.758197 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.758212 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:55:56.768277 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.758239 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 14:55:56.830378 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.830352 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-103.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 14:55:56.831472 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.830359 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-103.us-west-1.compute.internal.18e4423741400498 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-103.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-103.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 14:55:56.685649048 +0000 UTC m=+0.354936136,LastTimestamp:2026-11-02 14:55:56.685649048 +0000 UTC m=+0.354936136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-103.us-west-1.compute.internal,}" Nov 02 14:55:56.853141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.852997 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.853974 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.853953 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.854033 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.854000 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.854033 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.854011 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.854093 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.854038 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.858611 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.858592 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal"] Nov 02 14:55:56.858667 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.858655 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.859310 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.859290 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.859405 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.859320 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.859405 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.859334 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.861474 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.861459 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.862101 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.862177 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862104 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.862235 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862187 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.862235 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862218 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.862235 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862233 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.862767 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862752 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.862837 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862783 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.862837 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.862813 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.866397 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.866379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.866485 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.866401 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:56.867028 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.867011 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:56.867103 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.867043 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:56.867103 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.867058 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:56.878570 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.878547 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-103.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 14:55:56.895173 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.895149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.895246 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.895176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.895246 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.895195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.902976 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.902956 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.906692 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.906676 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.922908 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.922885 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 14:55:56.941629 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.941607 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:55:56.941703 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.941671 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bhjmm" Nov 02 14:55:56.970645 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.970617 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 14:55:56.970720 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.970606 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-103.us-west-1.compute.internal.18e4423742c6d35c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-103.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-103.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-103.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 14:55:56.71126102 +0000 UTC m=+0.380548111,LastTimestamp:2026-11-02 14:55:56.71126102 +0000 UTC m=+0.380548111,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-103.us-west-1.compute.internal,}" Nov 02 14:55:56.970720 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.970686 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 14:55:56.988649 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.988624 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-103.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.988715 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:56.988657 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-103.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 14:55:56.995625 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.995683 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.995683 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.995683 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.995810 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:56.995810 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:56.995709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.182433 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.182361 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-103.us-west-1.compute.internal\" not found" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.189577 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.189565 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:57.190474 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.190456 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:57.190580 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.190490 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:57.190580 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.190504 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:57.190580 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.190537 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.206115 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.206097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.210757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.210469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.246167 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.246153 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:57.246245 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.246172 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-103.us-west-1.compute.internal\": node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.363294 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.363272 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.463501 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.463480 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.563989 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.563963 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.632697 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.632666 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 14:55:57.633108 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.632819 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a65af20ceb8e0453d8b17c9c993c7f5c-df77204d0ad5c075.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-103.us-west-1.compute.internal.18e4423742c74e64\": read tcp 10.0.135.103:59554->3.136.250.228:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-103.us-west-1.compute.internal.18e4423742c74e64 default 5165 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-103.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-103.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-103.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 14:55:56 +0000 UTC,LastTimestamp:2026-11-02 14:55:56.854014975 +0000 UTC m=+0.523302063,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-103.us-west-1.compute.internal,}" Nov 02 14:55:57.664944 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.664921 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.765462 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.765415 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.866070 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.866045 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:57.930937 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.930916 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:55:57.931774 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.931758 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:55:57.931889 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.931790 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:55:57.931889 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.931816 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:55:57.946044 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.946005 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:56 +0000 UTC" deadline="2028-08-14 14:00:36.543286268 +0000 UTC" Nov 02 14:55:57.946044 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:57.946040 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15623h4m38.597248973s" Nov 02 14:55:57.966283 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:57.966265 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:55:58.023763 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.023707 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:55:58.023875 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.023768 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:55:58.023875 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.023812 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:55:58.093244 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.093222 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:58.148647 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.148627 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:55:58.149509 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.149495 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:55:58.203400 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.203382 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:55:58.230758 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:58.230734 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea26f2febf8f66e888f887686e2c7f4.slice/crio-1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb WatchSource:0}: Error finding container 1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb: Status 404 returned error can't find the container with id 1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb Nov 02 14:55:58.230939 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:58.230926 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f06372767f35a0b77d37ba2e829e38f.slice/crio-ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8 WatchSource:0}: Error finding container ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8: Status 404 returned error can't find the container with id ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8 Nov 02 14:55:58.234069 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.234051 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:55:58.358764 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.358701 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:55:58.671922 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.671848 2579 apiserver.go:52] "Watching apiserver" Nov 02 14:55:58.693776 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.693754 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 14:55:58.763011 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.762963 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"5f06372767f35a0b77d37ba2e829e38f","Type":"ContainerStarted","Data":"ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8"} Nov 02 14:55:58.763917 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.763897 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerStarted","Data":"1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb"} Nov 02 14:55:58.765989 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.765975 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:55:58.766244 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.766230 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-wpqbc","kube-system/konnectivity-agent-lw49d","openshift-image-registry/node-ca-bzr9k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-gffg7","openshift-network-diagnostics/network-check-target-wx98q","openshift-network-operator/iptables-alerter-b5jgk","kube-system/global-pull-secret-syncer-tzjrs","kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92","openshift-multus/multus-65hmq","openshift-multus/multus-additional-cni-plugins-5lqlt","openshift-dns/node-resolver-7x9dz","openshift-multus/network-metrics-daemon-245v8"] Nov 02 14:55:58.769684 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.769670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.771560 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.771545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.773519 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.773503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.775576 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.775560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.775632 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.775619 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:55:58.777634 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.777612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.780127 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.780114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:55:58.780199 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.780178 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:55:58.782202 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.782187 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.782267 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.782245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:55:58.784171 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.784157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.786247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.786232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.788440 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.788427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.790701 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.790686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.793000 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.792983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.794327 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.794315 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:55:58.807155 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.807229 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.807263 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.807263 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.807317 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807317 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807317 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.807423 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.807423 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807423 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2bz\" (UniqueName: \"kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807535 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.807535 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rj9g\" (UniqueName: \"kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807599 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.807599 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807599 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807599 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807759 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807759 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807759 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807759 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkq7b\" (UniqueName: \"kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9pd\" (UniqueName: \"kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.807933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.807985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808247 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.808640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp62d\" (UniqueName: \"kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwsp\" (UniqueName: \"kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.808970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809134 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbgl\" (UniqueName: \"kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5xs\" (UniqueName: \"kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.809652 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.809446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.817860 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.817842 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.817954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.817880 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.817954 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.817880 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:55:58.818043 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.817956 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snsjj\"" Nov 02 14:55:58.819129 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.819113 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.819222 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.819184 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrdxh\"" Nov 02 14:55:58.819305 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.819293 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.821598 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.821584 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.821673 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.821598 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.821673 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.821615 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:55:58.825236 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.825222 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4lkk\"" Nov 02 14:55:58.830338 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.828518 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zct6j\"" Nov 02 14:55:58.861381 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.861363 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:55:58.861444 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.861413 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhmkr" Nov 02 14:55:58.909672 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909650 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909711 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq7b\" (UniqueName: \"kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909840 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.909835 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909869 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909878 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909891 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909911 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pd\" (UniqueName: \"kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.909974 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909998 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.909974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.910055 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.410014181 +0000 UTC m=+3.079301274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910179 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.910534 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910216 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910246 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910272 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910258 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5lvgl\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910272 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dhf26\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910294 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910202 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910314 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910313 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910336 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910361 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910412 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910414 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5w9z9\"" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910436 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.911407 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910456 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zp9vx\"" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910452 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dn6m\"" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910477 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910525 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.910507 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910574 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.910640 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.41062426 +0000 UTC m=+3.079911350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910665 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.912480 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp62d\" (UniqueName: \"kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910834 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.910995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911093 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913085 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwsp\" (UniqueName: \"kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911378 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.913757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911460 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911545 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911630 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911683 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.914260 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbgl\" (UniqueName: \"kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5xs\" (UniqueName: \"kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911868 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911979 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.911982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.914668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912169 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912235 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2bz\" (UniqueName: \"kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rj9g\" (UniqueName: \"kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912436 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.915125 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912494 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912496 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.912544 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.913274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.913333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.913530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.913627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.914019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.914039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.915538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.914394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.920594 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.920579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:58.920700 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.920682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.921862 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.921845 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:58.963812 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.963769 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:55:58.963923 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.963816 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:55:58.963923 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.963829 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:55:58.963923 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:58.963891 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.46387306 +0000 UTC m=+3.133160150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:55:58.965908 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.965883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq7b\" (UniqueName: \"kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:58.965908 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.965896 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp62d\" (UniqueName: \"kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:58.966025 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.965928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9pd\" (UniqueName: \"kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:58.966170 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.966155 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwsp\" (UniqueName: \"kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:58.969900 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.969881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5xs\" (UniqueName: \"kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:58.970412 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.970393 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbgl\" (UniqueName: \"kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:58.971378 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.971361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2bz\" (UniqueName: \"kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:58.972719 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.972699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:58.973382 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:58.973368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rj9g\" (UniqueName: \"kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:55:59.085630 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.085119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:55:59.087832 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.087755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:55:59.093237 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.093068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:55:59.096378 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:59.096352 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ba3151_5436_402c_87c6_47dbf8fbb3ce.slice/crio-32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19 WatchSource:0}: Error finding container 32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19: Status 404 returned error can't find the container with id 32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19 Nov 02 14:55:59.097103 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.097084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:55:59.102508 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.102465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:55:59.105095 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:59.103827 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0959e584_488f_4332_a378_b927ac207e49.slice/crio-dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c WatchSource:0}: Error finding container dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c: Status 404 returned error can't find the container with id dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c Nov 02 14:55:59.107708 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.107689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:55:59.111633 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:59.111608 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91058aad_5292_4b62_ab2e_91444f80c847.slice/crio-46b9eec966c911123ee118a967b8010c19514b4a9c2a5847b345ac76318ff8c2 WatchSource:0}: Error finding container 46b9eec966c911123ee118a967b8010c19514b4a9c2a5847b345ac76318ff8c2: Status 404 returned error can't find the container with id 46b9eec966c911123ee118a967b8010c19514b4a9c2a5847b345ac76318ff8c2 Nov 02 14:55:59.121744 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.121721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65hmq" Nov 02 14:55:59.132079 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:59.132056 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8d1f65_2c06_4241_a3b4_025039c6e9e1.slice/crio-31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad WatchSource:0}: Error finding container 31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad: Status 404 returned error can't find the container with id 31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad Nov 02 14:55:59.133031 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.133011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:55:59.140186 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.140166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:55:59.143396 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:55:59.143375 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46626844_9af4_499e_8f53_7428a1deb452.slice/crio-a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30 WatchSource:0}: Error finding container a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30: Status 404 returned error can't find the container with id a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30 Nov 02 14:55:59.416246 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.416167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:55:59.416246 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.416228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:55:59.416473 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.416360 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:55:59.416473 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.416417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:00.416400281 +0000 UTC m=+4.085687358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:55:59.416585 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.416474 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:55:59.416585 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.416506 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:56:00.416496339 +0000 UTC m=+4.085783418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:55:59.517235 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.517195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:55:59.517408 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.517349 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:55:59.517408 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.517371 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:55:59.517408 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.517386 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:55:59.517553 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:55:59.517440 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:00.517422731 +0000 UTC m=+4.186709823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:55:59.767362 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.767313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lw49d" event={"ID":"7734be1e-a6d2-4c25-9982-3f0fbe90c330","Type":"ContainerStarted","Data":"9da91326d1f7a53509cf2745cc1513c4677894ca82b893902ad0ac19230df37b"} Nov 02 14:55:59.768785 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.768757 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzr9k" event={"ID":"0959e584-488f-4332-a378-b927ac207e49","Type":"ContainerStarted","Data":"dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c"} Nov 02 14:55:59.770066 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.770031 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" event={"ID":"02ba3151-5436-402c-87c6-47dbf8fbb3ce","Type":"ContainerStarted","Data":"32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19"} Nov 02 14:55:59.771391 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.771360 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerStarted","Data":"11fa4f46fad1439ecdeb0044c7282a9970a12d1e2558729678d132909b6a07d2"} Nov 02 14:55:59.772600 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.772577 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30"} Nov 02 14:55:59.773911 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.773867 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65hmq" event={"ID":"5b8d1f65-2c06-4241-a3b4-025039c6e9e1","Type":"ContainerStarted","Data":"31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad"} Nov 02 14:55:59.775130 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.775090 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5jgk" event={"ID":"6fc4fca4-e664-4419-9ad8-a6930c959473","Type":"ContainerStarted","Data":"5f6d84dc13e9f38c205020d35bdfe16b1b8601fad82731275b29544ea85f1244"} Nov 02 14:55:59.776266 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.776241 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gffg7" event={"ID":"91058aad-5292-4b62-ab2e-91444f80c847","Type":"ContainerStarted","Data":"46b9eec966c911123ee118a967b8010c19514b4a9c2a5847b345ac76318ff8c2"} Nov 02 14:55:59.777466 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.777448 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7x9dz" event={"ID":"2cdb8471-d93e-4750-9aef-984d1a738d4a","Type":"ContainerStarted","Data":"f8e526a2cb011b53caf302e966730dc2a9b8f35455c911ed979f690ac4b1690d"} Nov 02 14:55:59.864478 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.864419 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:58 +0000 UTC" deadline="2028-05-30 14:55:24.965069277 +0000 UTC" Nov 02 14:55:59.864478 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:55:59.864473 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13799h59m25.100600647s" Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.426022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.426110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.426153 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.426218 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.426230 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.426209509 +0000 UTC m=+6.095496586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:00.426567 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.426266 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.426250976 +0000 UTC m=+6.095538068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:00.526703 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.526418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:00.526703 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.526584 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:00.526703 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.526605 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:00.526703 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.526618 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:00.526703 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.526666 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.526650041 +0000 UTC m=+6.195937131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:00.759177 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.758907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:00.759353 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.758907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:00.759353 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.759314 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:00.759459 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.759372 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:00.759459 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.758995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:00.759546 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:00.759482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:00.808059 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.807366 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"5f06372767f35a0b77d37ba2e829e38f","Type":"ContainerStarted","Data":"a5dbdd3512bb06d5f7df705bd12f6afc533a56cc6563b28ff5115a689ede80e4"} Nov 02 14:56:00.817298 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.816630 2579 generic.go:356] "Generic (PLEG): container finished" podID="aea26f2febf8f66e888f887686e2c7f4" containerID="39e2ea0aa2c18cdc40da410f042f02f4a1060d0d4e21816c409d058ceb59bd81" exitCode=0 Nov 02 14:56:00.817298 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.816784 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerDied","Data":"39e2ea0aa2c18cdc40da410f042f02f4a1060d0d4e21816c409d058ceb59bd81"} Nov 02 14:56:00.864888 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.864791 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:58 +0000 UTC" deadline="2028-07-11 17:06:59.434893656 +0000 UTC" Nov 02 14:56:00.864888 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:00.864861 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14810h10m58.570036046s" Nov 02 14:56:01.037222 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:01.037166 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" podStartSLOduration=3.037145451 podStartE2EDuration="3.037145451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:00.921959821 +0000 UTC m=+4.591246920" watchObservedRunningTime="2026-11-02 14:56:01.037145451 +0000 UTC m=+4.706432550" Nov 02 14:56:01.838113 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:01.837112 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerStarted","Data":"d14cfb9e8750143fafe53134fdb474c8592e52e621e4acaaf552c57b36b31c06"} Nov 02 14:56:01.957105 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:01.957047 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" podStartSLOduration=3.957028111 podStartE2EDuration="3.957028111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:01.956436145 +0000 UTC m=+5.625723262" watchObservedRunningTime="2026-11-02 14:56:01.957028111 +0000 UTC m=+5.626315208" Nov 02 14:56:02.441633 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.441600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:02.441845 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.441662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:02.441845 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.441813 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:02.441962 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.441874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:06.441855118 +0000 UTC m=+10.111142193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:02.441962 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.441904 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:02.441962 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.441933 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:56:06.44192516 +0000 UTC m=+10.111212248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:02.542454 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.542132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:02.542454 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.542322 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:02.542454 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.542342 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:02.542454 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.542354 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:02.542454 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.542413 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:06.542396169 +0000 UTC m=+10.211683260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:02.760612 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.760539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:02.760768 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.760650 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:02.761089 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.760985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:02.761165 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.761087 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:02.761165 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.761134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:02.761263 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:02.761199 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:02.841185 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.841125 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="8a4889e4786c8d5eae8eef0c4adb0d133762100371b06aec6a8e8433122dec58" exitCode=0 Nov 02 14:56:02.841935 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:02.841904 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"8a4889e4786c8d5eae8eef0c4adb0d133762100371b06aec6a8e8433122dec58"} Nov 02 14:56:03.852229 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:03.852179 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lw49d" event={"ID":"7734be1e-a6d2-4c25-9982-3f0fbe90c330","Type":"ContainerStarted","Data":"33a9f15a2d1a029483757c67da3a72e8cbca2bf7b84ebad4b724b5494d65ebbb"} Nov 02 14:56:04.761575 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:04.761545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:04.761721 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:04.761683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:04.761778 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:04.761765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:04.761885 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:04.761868 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:04.761952 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:04.761923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:04.762006 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:04.761992 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:04.860080 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:04.860026 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzr9k" event={"ID":"0959e584-488f-4332-a378-b927ac207e49","Type":"ContainerStarted","Data":"b7353df6436f687cf27522b448c22af90fd6d7edb5d916d11f79a8275db29b79"} Nov 02 14:56:04.975781 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:04.975719 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lw49d" podStartSLOduration=3.706494378 podStartE2EDuration="7.975699578s" totalImagesPullingTime="4.2692052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:03.969810883 +0000 UTC m=+7.639097973" watchObservedRunningTime="2026-11-02 14:56:04.975699578 +0000 UTC m=+8.644986676" Nov 02 14:56:05.866637 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:05.866600 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" event={"ID":"02ba3151-5436-402c-87c6-47dbf8fbb3ce","Type":"ContainerStarted","Data":"90c286eef7102a14e7865391adb73ec933d3cc987719c22cfccf375d933cc5e7"} Nov 02 14:56:06.475668 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.475635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:06.475822 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.475699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:06.475876 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.475860 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:06.475936 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.475926 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:14.475907017 +0000 UTC m=+18.145194115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:06.476382 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.476303 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:06.476382 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.476353 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:56:14.476339194 +0000 UTC m=+18.145626274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:06.576513 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.576221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:06.576513 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.576393 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:06.576513 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.576411 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:06.576513 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.576423 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:06.576513 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.576476 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:14.576458297 +0000 UTC m=+18.245745388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.777656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.777902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.778449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.778659 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:06.779026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:06.782321 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:06.779366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:07.124770 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.124704 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:07.125252 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.125235 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:07.246657 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.246384 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzr9k" podStartSLOduration=5.053694614 podStartE2EDuration="10.246365597s" totalImagesPullingTime="5.192670983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:04.976499658 +0000 UTC m=+8.645786756" watchObservedRunningTime="2026-11-02 14:56:07.246365597 +0000 UTC m=+10.915652695" Nov 02 14:56:07.389877 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.389842 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:56:07.759029 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.758708 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:07.38986415Z","UUID":"0cbc2747-b1db-4ac7-99a5-77bf82448d72","Handler":null,"Name":"","Endpoint":""} Nov 02 14:56:07.760427 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.760395 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:56:07.760427 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.760427 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:56:07.873902 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.873867 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="c0a97e687d9501b585b036dc8b4df7dc97b16c5f919cc1181864708c1f1fe30f" exitCode=0 Nov 02 14:56:07.874059 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.873953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"c0a97e687d9501b585b036dc8b4df7dc97b16c5f919cc1181864708c1f1fe30f"} Nov 02 14:56:07.876640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.876513 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65hmq" event={"ID":"5b8d1f65-2c06-4241-a3b4-025039c6e9e1","Type":"ContainerStarted","Data":"d7c7897829a65d558e76a0133aada02e8ecf3ab8b9998b49cfe4514f7d5aac9a"} Nov 02 14:56:07.878355 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.878335 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7x9dz" event={"ID":"2cdb8471-d93e-4750-9aef-984d1a738d4a","Type":"ContainerStarted","Data":"7cb22c5ceeb67c89ac65a839438f108174272fbb9864cc188033075db124a783"} Nov 02 14:56:07.880154 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:07.880132 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" event={"ID":"02ba3151-5436-402c-87c6-47dbf8fbb3ce","Type":"ContainerStarted","Data":"55aed7541fd9813e71727330c5c9fd31db00a10de9bd5a81946bfb0700221f21"} Nov 02 14:56:08.131518 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.131457 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-65hmq" podStartSLOduration=2.956288011 podStartE2EDuration="11.131437656s" totalImagesPullingTime="8.175149645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:08.131326543 +0000 UTC m=+11.800613641" watchObservedRunningTime="2026-11-02 14:56:08.131437656 +0000 UTC m=+11.800724755" Nov 02 14:56:08.254986 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.254925 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7x9dz" podStartSLOduration=3.052563582 podStartE2EDuration="11.254904958s" totalImagesPullingTime="8.202341376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:08.254562613 +0000 UTC m=+11.923849712" watchObservedRunningTime="2026-11-02 14:56:08.254904958 +0000 UTC m=+11.924192058" Nov 02 14:56:08.759452 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.759419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:08.759618 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:08.759553 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:08.759618 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.759565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:08.759744 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:08.759643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:08.759744 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.759663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:08.759744 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:08.759735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:08.886669 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:08.886640 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5jgk" event={"ID":"6fc4fca4-e664-4419-9ad8-a6930c959473","Type":"ContainerStarted","Data":"5c4f1dd923b34ff0953665c2cebaa1dd7ea756391a625f8686bcf9e6763125d3"} Nov 02 14:56:09.889970 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:09.889921 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" event={"ID":"02ba3151-5436-402c-87c6-47dbf8fbb3ce","Type":"ContainerStarted","Data":"0cf3903b6633abd4dfd1a6329b34f3a7da6b560f76f28320257654a7243a2060"} Nov 02 14:56:09.892150 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:09.892127 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="53f0b142c38efcac971a5f9d64544bfc6cb6b15e910be3e3ceb169114c3b18cc" exitCode=0 Nov 02 14:56:09.892275 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:09.892201 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"53f0b142c38efcac971a5f9d64544bfc6cb6b15e910be3e3ceb169114c3b18cc"} Nov 02 14:56:09.894072 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:09.893984 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gffg7" event={"ID":"91058aad-5292-4b62-ab2e-91444f80c847","Type":"ContainerStarted","Data":"82d910076fd06d79982427784002f49e54a8dbd7733b68bf8a57943428b8b937"} Nov 02 14:56:10.007957 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.007906 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b5jgk" podStartSLOduration=4.889290665 podStartE2EDuration="13.007886655s" totalImagesPullingTime="8.11859599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:09.004871446 +0000 UTC m=+12.674158544" watchObservedRunningTime="2026-11-02 14:56:10.007886655 +0000 UTC m=+13.677173753" Nov 02 14:56:10.008103 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.008042 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" podStartSLOduration=3.6308543 podStartE2EDuration="13.008036122s" totalImagesPullingTime="9.377181822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:10.007776935 +0000 UTC m=+13.677064033" watchObservedRunningTime="2026-11-02 14:56:10.008036122 +0000 UTC m=+13.677323220" Nov 02 14:56:10.248086 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.248023 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gffg7" podStartSLOduration=3.061719462 podStartE2EDuration="13.248007485s" totalImagesPullingTime="10.186288023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:10.24747271 +0000 UTC m=+13.916759808" watchObservedRunningTime="2026-11-02 14:56:10.248007485 +0000 UTC m=+13.917294619" Nov 02 14:56:10.761975 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.761944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:10.761975 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.761972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:10.762200 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:10.762070 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:10.762200 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:10.762140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:10.762200 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:10.762169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:10.762332 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:10.762267 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:11.900287 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:11.900092 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="8fcbeaef1a14ece725300cdd6ff385e24c8d40e93a1e931f3409717ab47b48a6" exitCode=0 Nov 02 14:56:11.900287 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:11.900161 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"8fcbeaef1a14ece725300cdd6ff385e24c8d40e93a1e931f3409717ab47b48a6"} Nov 02 14:56:12.093316 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.093290 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:12.093470 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.093443 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:12.094204 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.094179 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:12.761916 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.761889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:12.762083 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.761920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:12.762083 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:12.761920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:12.762083 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:12.762005 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:12.762218 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:12.762114 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:12.762218 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:12.762183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:14.543559 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.543395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:14.543905 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.543553 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:14.543905 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.543595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:14.543905 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.543614 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret podName:09413fd8-4cfb-4e13-adb8-c8a58533ad5e nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.543596052 +0000 UTC m=+34.212883133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret") pod "global-pull-secret-syncer-tzjrs" (UID: "09413fd8-4cfb-4e13-adb8-c8a58533ad5e") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:14.543905 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.543659 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:14.543905 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.543701 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.54368982 +0000 UTC m=+34.212976896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:14.644847 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.644816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:14.645023 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.645002 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:14.645073 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.645037 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:14.645073 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.645051 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k7lvk for pod openshift-network-diagnostics/network-check-target-wx98q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:14.645142 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.645115 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk podName:ab3ac100-fe63-4c64-8587-6c137b20f5d5 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.64509348 +0000 UTC m=+34.314380564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7lvk" (UniqueName: "kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk") pod "network-check-target-wx98q" (UID: "ab3ac100-fe63-4c64-8587-6c137b20f5d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:14.758531 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.758499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:14.758698 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.758538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:14.758698 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.758508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:14.758698 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.758619 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:14.758835 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.758695 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:14.758835 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:14.758757 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:14.908544 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.908509 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="3398679e405883a7f2ecfa8c556d2ef06c217bb2586c675c50888c77b662bcce" exitCode=0 Nov 02 14:56:14.908722 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.908600 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"3398679e405883a7f2ecfa8c556d2ef06c217bb2586c675c50888c77b662bcce"} Nov 02 14:56:14.911556 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911525 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"5c35604a96b7ad4bbe1b17b1f5778e8e9a4f8fc11364379567d49c315ff0095b"} Nov 02 14:56:14.911640 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911591 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"aee797813d9a3660b8d9b32fd0ae23925366089463471a653fd68045d1947b37"} Nov 02 14:56:14.911676 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911621 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"14d1c3cd714c95db8baf5eea569bb64808792fa22a88328d48c76f963b611b31"} Nov 02 14:56:14.911711 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911685 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"8ffeb6cf266160db5b4f3d012fce2a531dac980d4e1ec32b235cd7d0fb5b4661"} Nov 02 14:56:14.911745 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911711 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"a66baecac5848adf236a4689841a0923a2af5ec906f2978bd46919fd28a139cc"} Nov 02 14:56:14.911745 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:14.911739 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"7d6fffa4cda0e1f975dfb2066162d878acf54e3ed0f1e173ccfc233ebeb44fbf"} Nov 02 14:56:15.916509 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:15.916474 2579 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="4718ed82e63c8de270a4bf0ebfdb6097c3797dbe492fd652250cf7579ea098d0" exitCode=0 Nov 02 14:56:15.916981 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:15.916546 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerDied","Data":"4718ed82e63c8de270a4bf0ebfdb6097c3797dbe492fd652250cf7579ea098d0"} Nov 02 14:56:16.759246 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:16.759213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:16.759413 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:16.759350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:16.759413 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:16.759347 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:16.759525 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:16.759442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:16.759525 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:16.759455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:16.759632 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:16.759533 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:16.922061 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:16.921981 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" event={"ID":"51dfa937-57ef-4edc-b10d-2d995c93e764","Type":"ContainerStarted","Data":"17f93eaa635dbba1138a58b5f5a7e898f490ce4d9bc30feba30eb3ae95e602ea"} Nov 02 14:56:16.925083 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:16.925065 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"f54893d565643a6c5e3770fec5e682f40c602ace19b90a4114f813d759d6e851"} Nov 02 14:56:17.077121 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:17.077079 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5lqlt" podStartSLOduration=7.566592682 podStartE2EDuration="20.077066795s" totalImagesPullingTime="12.510474113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:17.076524567 +0000 UTC m=+20.745811664" watchObservedRunningTime="2026-11-02 14:56:17.077066795 +0000 UTC m=+20.746353892" Nov 02 14:56:18.758770 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.758745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:18.758770 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.758754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:18.759402 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:18.758909 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:18.759402 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.758918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:18.759402 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:18.759017 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:18.759402 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:18.759113 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:18.933376 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.933205 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" event={"ID":"46626844-9af4-499e-8f53-7428a1deb452","Type":"ContainerStarted","Data":"72843375dd44bb5057952f7f740917e5f3659f1ef75aa155665029db113765f5"} Nov 02 14:56:18.933541 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.933425 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:18.948935 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:18.948880 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:19.211614 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:19.211574 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" podStartSLOduration=7.16108105 podStartE2EDuration="22.21156123s" totalImagesPullingTime="15.05048018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:19.211145963 +0000 UTC m=+22.880433060" watchObservedRunningTime="2026-11-02 14:56:19.21156123 +0000 UTC m=+22.880848327" Nov 02 14:56:19.937840 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:19.936944 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:19.938931 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:19.938910 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:19.956919 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:19.956892 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:20.758832 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:20.758788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:20.759078 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:20.758941 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:20.759078 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:20.758952 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:20.759078 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:20.759036 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:20.759189 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:20.759094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:20.759189 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:20.759172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:20.939397 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:20.939368 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:21.447137 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.447061 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzjrs"] Nov 02 14:56:21.447296 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.447175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:21.447296 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:21.447282 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:21.450725 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.450695 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx98q"] Nov 02 14:56:21.450867 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.450789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:21.450954 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:21.450932 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:21.451476 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.451454 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-245v8"] Nov 02 14:56:21.451611 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.451590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:21.451724 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:21.451703 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:21.935788 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.935755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:21.941061 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:21.941042 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:22.249757 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:22.249728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:22.474627 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:22.474595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:22.758632 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:22.758603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:22.758820 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:22.758641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:22.758820 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:22.758729 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:22.758947 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:22.758883 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:23.758861 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:23.758775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:23.759223 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:23.758888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:24.082850 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.082759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:24.171870 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.171837 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:24.172040 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.172027 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:24.182795 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.182762 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" podUID="46626844-9af4-499e-8f53-7428a1deb452" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 14:56:24.190831 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.190793 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" podUID="46626844-9af4-499e-8f53-7428a1deb452" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 14:56:24.298602 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.298575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:24.759223 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.759189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:24.759679 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:24.759194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:24.759679 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:24.759293 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:24.759679 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:24.759352 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:25.759083 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:25.759047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:25.759248 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:25.759153 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzjrs" podUID="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" Nov 02 14:56:25.860786 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:25.860753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:26.073076 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:26.073000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:26.521674 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:26.521640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:26.736643 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:26.736491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:26.759182 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:26.759156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:26.759286 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:26.759268 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245v8" podUID="50dc24f3-a26e-4583-bfe3-f1a25e97315a" Nov 02 14:56:26.759601 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:26.759343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:26.759601 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:26.759459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx98q" podUID="ab3ac100-fe63-4c64-8587-6c137b20f5d5" Nov 02 14:56:27.108141 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.108071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:27.115570 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.115549 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeReady" Nov 02 14:56:27.115703 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.115693 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 14:56:27.232124 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.232098 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w7g64"] Nov 02 14:56:27.238559 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.238540 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vxl2w"] Nov 02 14:56:27.238684 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.238647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.242524 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.242506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.291907 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.291886 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:56:27.292174 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.292156 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:56:27.292256 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.292188 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bm8rt\"" Nov 02 14:56:27.295005 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.294981 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:56:27.295076 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.295023 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:56:27.295076 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.295047 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wtdgk\"" Nov 02 14:56:27.309511 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.309484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:27.329438 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.329419 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:56:27.338666 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.338728 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.338728 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65c8\" (UniqueName: \"kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.338728 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c256h\" (UniqueName: \"kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.338859 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.338896 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.338884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.346300 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.346282 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vxl2w"] Nov 02 14:56:27.349917 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.349896 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7g64"] Nov 02 14:56:27.439907 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.439848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.439907 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.439882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65c8\" (UniqueName: \"kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.440088 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.439913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c256h\" (UniqueName: \"kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.440088 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:27.439989 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:27.440088 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:27.440048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:27.940033818 +0000 UTC m=+31.609320893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:27.440253 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.440150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.440253 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.440197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.440253 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.440236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.440529 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.440474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.440746 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.440727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.443999 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.443979 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.502965 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.502945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65c8\" (UniqueName: \"kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.503124 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.503107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c256h\" (UniqueName: \"kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.556502 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.556480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:27.759299 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.759275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:27.812241 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.812219 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:56:27.812352 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.812299 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dh847\"" Nov 02 14:56:27.945601 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.945567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:27.945747 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:27.945713 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:27.945819 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:27.945786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:28.945770251 +0000 UTC m=+32.615057327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:27.997351 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:27.997322 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vxl2w"] Nov 02 14:56:28.000184 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:56:28.000157 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51a2287_9459_4c81_8646_15515c5439d9.slice/crio-52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0 WatchSource:0}: Error finding container 52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0: Status 404 returned error can't find the container with id 52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0 Nov 02 14:56:28.758631 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.758591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:28.758631 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.758620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:28.811558 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.811536 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:56:28.811558 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.811546 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:56:28.860933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.860904 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:56:28.860933 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.860919 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nw765\"" Nov 02 14:56:28.861371 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.861358 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ptvxg\"" Nov 02 14:56:28.952508 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.952459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:28.952667 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:28.952635 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:28.952737 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:28.952700 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.952679655 +0000 UTC m=+34.621966736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:28.957337 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:28.957312 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vxl2w" event={"ID":"a51a2287-9459-4c81-8646-15515c5439d9","Type":"ContainerStarted","Data":"52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0"} Nov 02 14:56:29.526580 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.526545 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4prgg"] Nov 02 14:56:29.544918 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.544887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.608512 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.608482 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:56:29.608512 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.608501 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:56:29.655389 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.655362 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:56:29.655389 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.655363 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:56:29.655629 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.655602 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fbnz4\"" Nov 02 14:56:29.658936 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.658912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.659046 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.658948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.659108 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.659056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6l28\" (UniqueName: \"kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.659162 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.659110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.659162 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.659139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.666920 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.666897 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4prgg"] Nov 02 14:56:29.759628 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.759595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l28\" (UniqueName: \"kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.759866 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.759655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.759866 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.759679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.759866 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.759732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.759866 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.759755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.759866 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:29.759844 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:29.760116 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:29.759902 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.259883881 +0000 UTC m=+33.929170957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:29.760441 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.760418 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.760516 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:29.760455 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:29.760516 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:29.760515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.260498601 +0000 UTC m=+33.929785682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:29.760729 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.760706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.762979 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.762953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:29.788286 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.788227 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:56:29.822091 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.822064 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l28\" (UniqueName: \"kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:29.824952 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.824926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:29.864355 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.864254 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:56:29.890847 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.890814 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 14:56:29.891090 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.891073 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mnrgm\"" Nov 02 14:56:29.960693 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.960665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:29.960693 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.960697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:29.960901 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.960809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:29.960937 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.960919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:29.986759 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.986723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:29.992417 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.992393 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 14:56:29.993498 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:29.993478 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc"] Nov 02 14:56:30.020686 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.020653 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc"] Nov 02 14:56:30.020872 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.020781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:30.061649 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.061597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.061737 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.061651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.061737 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.061699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.061737 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.061720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.062127 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.062110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.062227 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.062212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.062387 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.062370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.105974 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.105938 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 14:56:30.106094 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.106056 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mcvdb\"" Nov 02 14:56:30.106094 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.106073 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:30.122684 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.122654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.153976 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.153947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:30.162191 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.162173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsqb\" (UniqueName: \"kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb\") pod \"migrator-66df5d496b-l69rc\" (UID: \"683f8569-4473-4d34-ae00-ba1bf3bf9f5d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:30.189254 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.189226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:30.263204 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.263175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:30.263367 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.263346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:30.263438 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.263352 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:30.263438 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.263389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsqb\" (UniqueName: \"kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb\") pod \"migrator-66df5d496b-l69rc\" (UID: \"683f8569-4473-4d34-ae00-ba1bf3bf9f5d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:30.263533 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.263390 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:30.263533 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.263473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:31.263458573 +0000 UTC m=+34.932745669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:30.263638 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.263547 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:31.263523964 +0000 UTC m=+34.932811042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:30.324838 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.324751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsqb\" (UniqueName: \"kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb\") pod \"migrator-66df5d496b-l69rc\" (UID: \"683f8569-4473-4d34-ae00-ba1bf3bf9f5d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:30.343730 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.343699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:30.399538 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.399502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:30.561966 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.561934 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:56:30.565325 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.565298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:30.565454 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.565433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:30.565494 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.565454 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:56:30.565559 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.565541 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:02.565502051 +0000 UTC m=+66.234789142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : secret "metrics-daemon-secret" not found Nov 02 14:56:30.565769 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:56:30.565747 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e14b273_2cc6_4872_88e3_fec55e2b0631.slice/crio-d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614 WatchSource:0}: Error finding container d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614: Status 404 returned error can't find the container with id d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614 Nov 02 14:56:30.567998 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.567976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:30.666063 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.665999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:30.668985 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.668947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:30.744901 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.744870 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc"] Nov 02 14:56:30.748130 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:56:30.748092 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683f8569_4473_4d34_ae00_ba1bf3bf9f5d.slice/crio-1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81 WatchSource:0}: Error finding container 1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81: Status 404 returned error can't find the container with id 1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81 Nov 02 14:56:30.776906 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.776879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:30.904086 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.903875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:30.963164 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.963134 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" event={"ID":"683f8569-4473-4d34-ae00-ba1bf3bf9f5d","Type":"ContainerStarted","Data":"1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81"} Nov 02 14:56:30.964303 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.964275 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" event={"ID":"8e14b273-2cc6-4872-88e3-fec55e2b0631","Type":"ContainerStarted","Data":"d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614"} Nov 02 14:56:30.969643 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:30.969620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:30.969770 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.969754 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:30.969846 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:30.969835 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.969817546 +0000 UTC m=+38.639104636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:31.188008 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.187982 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzjrs"] Nov 02 14:56:31.272284 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.272251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:31.272428 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.272307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:31.272428 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:31.272369 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:31.272428 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:31.272428 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:33.27241271 +0000 UTC m=+36.941699798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:31.272561 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:31.272425 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:31.272561 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:31.272455 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:33.27244907 +0000 UTC m=+36.941736175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:31.277330 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:56:31.277304 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09413fd8_4cfb_4e13_adb8_c8a58533ad5e.slice/crio-f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315 WatchSource:0}: Error finding container f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315: Status 404 returned error can't find the container with id f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315 Nov 02 14:56:31.304497 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.304474 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx98q"] Nov 02 14:56:31.307329 ip-10-0-135-103 kubenswrapper[2579]: W1102 14:56:31.307306 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3ac100_fe63_4c64_8587_6c137b20f5d5.slice/crio-a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01 WatchSource:0}: Error finding container a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01: Status 404 returned error can't find the container with id a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01 Nov 02 14:56:31.418057 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.418024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:31.625858 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.625762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:31.968279 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.968242 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzjrs" event={"ID":"09413fd8-4cfb-4e13-adb8-c8a58533ad5e","Type":"ContainerStarted","Data":"f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315"} Nov 02 14:56:31.969925 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.969897 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vxl2w" event={"ID":"a51a2287-9459-4c81-8646-15515c5439d9","Type":"ContainerStarted","Data":"6a4a24a9ea88a28e8b3d001b23150250be83965ce9feb3bc763151c7051875d5"} Nov 02 14:56:31.971522 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:31.971496 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx98q" event={"ID":"ab3ac100-fe63-4c64-8587-6c137b20f5d5","Type":"ContainerStarted","Data":"a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01"} Nov 02 14:56:32.093196 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:32.093146 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vxl2w" podStartSLOduration=1.7860599320000001 podStartE2EDuration="5.093128143s" totalImagesPullingTime="3.307068211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:32.092258964 +0000 UTC m=+35.761546063" watchObservedRunningTime="2026-11-02 14:56:32.093128143 +0000 UTC m=+35.762415245" Nov 02 14:56:32.558117 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:32.558031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:32.976702 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:32.976665 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" event={"ID":"683f8569-4473-4d34-ae00-ba1bf3bf9f5d","Type":"ContainerStarted","Data":"4d3859b0ec4c5847bed8fcff79f69a50ac859772e39ed240b6c6690092f89db3"} Nov 02 14:56:33.291190 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:33.291151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:33.291368 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:33.291220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:33.291368 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:33.291299 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:33.291368 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:33.291320 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:33.291368 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:33.291362 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.291346298 +0000 UTC m=+40.960633376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:33.291537 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:33.291385 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.291368478 +0000 UTC m=+40.960655572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:33.981331 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:33.981293 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" event={"ID":"8e14b273-2cc6-4872-88e3-fec55e2b0631","Type":"ContainerStarted","Data":"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97"} Nov 02 14:56:34.098108 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:34.098048 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" podStartSLOduration=2.065593432 podStartE2EDuration="5.098029503s" totalImagesPullingTime="3.032436071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:34.09713292 +0000 UTC m=+37.766420019" watchObservedRunningTime="2026-11-02 14:56:34.098029503 +0000 UTC m=+37.767316599" Nov 02 14:56:34.098291 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:34.098150 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" podStartSLOduration=3.077435139 podStartE2EDuration="5.098144669s" totalImagesPullingTime="2.02070953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:33.094975926 +0000 UTC m=+36.764263025" watchObservedRunningTime="2026-11-02 14:56:34.098144669 +0000 UTC m=+37.767431767" Nov 02 14:56:35.016897 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:35.007330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:35.016897 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:35.007481 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:35.016897 ip-10-0-135-103 kubenswrapper[2579]: E1102 14:56:35.007543 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:43.007523697 +0000 UTC m=+46.676810790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:35.987774 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:35.987739 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx98q" event={"ID":"ab3ac100-fe63-4c64-8587-6c137b20f5d5","Type":"ContainerStarted","Data":"98446aab8d70ccce08755881a0e4bb0e2f9a29e73b186d49f00de63b0dd725e5"} Nov 02 14:56:35.987968 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:35.987891 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:36.104642 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:36.104524 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wx98q" podStartSLOduration=34.667061209 podStartE2EDuration="39.104504862s" totalImagesPullingTime="4.437443653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:36.103595105 +0000 UTC m=+39.772882204" watchObservedRunningTime="2026-11-02 14:56:36.104504862 +0000 UTC m=+39.773791963" Nov 02 14:56:36.412475 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:36.412409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:36.994766 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:36.994722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzjrs" event={"ID":"09413fd8-4cfb-4e13-adb8-c8a58533ad5e","Type":"ContainerStarted","Data":"e16958ccd68193df7156f87e37e3f2aa56f50bfa860e977998288563989e4a4a"} Nov 02 14:56:37.058755 ip-10-0-135-103 kubenswrapper[2579]: I1102 14:56:37.058396 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:56:37.058552 ip-10-0-135-103 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 14:56:37.068214 ip-10-0-135-103 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 14:56:37.068394 ip-10-0-135-103 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 14:56:37.068440 ip-10-0-135-103 systemd[1]: kubelet.service: Consumed 2.362s CPU time. Nov 02 14:56:37.078389 ip-10-0-135-103 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:56:37.144298 ip-10-0-135-103 kubenswrapper[4469]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:37.144298 ip-10-0-135-103 kubenswrapper[4469]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:56:37.144298 ip-10-0-135-103 kubenswrapper[4469]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:37.144298 ip-10-0-135-103 kubenswrapper[4469]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:37.146329 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146256 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.146329 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146329 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.146329 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146332 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146336 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146340 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146344 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146349 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146353 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146357 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146361 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146365 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146368 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146370 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146373 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146375 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146378 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146381 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146384 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146386 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146389 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146391 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.146424 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146394 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146397 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146399 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146402 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146405 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146407 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146410 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146413 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146416 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146426 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146430 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146433 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146436 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146438 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146441 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146444 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146446 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146479 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146485 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.146893 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146490 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146493 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146496 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146499 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146501 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146504 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146507 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146509 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146512 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146515 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146518 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146521 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146523 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146526 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146529 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146532 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146535 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146539 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146541 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146544 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.147347 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146547 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146550 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146552 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146555 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146558 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146560 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146563 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146566 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146569 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146572 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146574 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146577 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146579 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146582 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146584 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146587 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146589 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146593 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146595 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146599 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.147863 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146602 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146605 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146608 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146611 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146613 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146616 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146618 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146621 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146624 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146627 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146629 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146632 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146634 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146637 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146640 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146642 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146645 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146647 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146650 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146652 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.148391 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146655 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146658 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146660 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146663 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146666 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.146671 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147452 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147460 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147463 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147467 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147470 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147473 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147476 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147479 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147482 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147484 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147487 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147491 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147495 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.148897 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147500 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147504 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147506 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147509 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147512 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147515 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147517 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147520 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147523 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147525 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147528 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147530 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147533 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147536 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147539 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147541 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147544 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147547 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147550 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147554 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.149368 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147558 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147561 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147563 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147566 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147569 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147572 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147574 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147577 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147580 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147583 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147585 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147589 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147591 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147594 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147597 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147599 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147602 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147605 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147607 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.149896 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147610 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147612 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147616 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147618 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147621 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147624 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147627 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147630 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147632 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147635 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147638 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147641 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147643 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147646 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147649 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147651 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147654 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147657 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147659 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147662 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.150369 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147664 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147667 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147670 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147672 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147675 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147677 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147680 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147682 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147685 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147687 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147690 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147692 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147695 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147698 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147700 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147703 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147705 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147708 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147711 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147714 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.150867 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147716 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147719 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147722 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147724 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147727 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147730 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147733 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147735 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147738 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147740 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147743 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147746 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147748 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.147752 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148258 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148264 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148267 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148270 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148273 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.151349 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148275 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148278 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148281 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148284 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148287 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148289 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148292 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148295 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148297 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148300 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148303 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148306 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148309 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148312 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148314 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148317 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148320 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148323 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148325 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.151821 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148328 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148331 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148333 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148336 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148339 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148341 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148344 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148347 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148350 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148358 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148361 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148364 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148366 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148369 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148371 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148374 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148377 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148380 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148382 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148385 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148387 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.152283 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148390 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148393 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148396 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148399 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148401 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148404 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148407 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148409 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148412 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148414 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148417 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148419 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148422 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148424 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148428 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148431 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148434 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148437 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148440 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.152791 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148443 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148446 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148449 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148451 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148454 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148456 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148460 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148462 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148465 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148468 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148471 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148473 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148476 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148478 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148481 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148483 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148486 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148489 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148491 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.153269 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148494 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148497 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148499 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148502 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148505 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148507 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148510 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148512 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148515 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148517 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148520 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148523 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148525 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148528 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148530 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148536 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148539 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148541 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148544 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.153773 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148547 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148550 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148555 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.148558 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148653 4469 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148667 4469 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148680 4469 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148685 4469 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148689 4469 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148693 4469 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148697 4469 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148705 4469 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148708 4469 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148711 4469 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148715 4469 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148718 4469 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148721 4469 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148728 4469 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148731 4469 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148734 4469 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148737 4469 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148740 4469 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148749 4469 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:56:37.154257 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148752 4469 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148755 4469 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148758 4469 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148762 4469 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148766 4469 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148769 4469 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148778 4469 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148782 4469 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148785 4469 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148788 4469 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148791 4469 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148794 4469 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148810 4469 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148821 4469 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148824 4469 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148829 4469 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148832 4469 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148835 4469 flags.go:64] FLAG: --enable-server="true" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148838 4469 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148846 4469 flags.go:64] FLAG: --event-burst="100" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148850 4469 flags.go:64] FLAG: --event-qps="50" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148853 4469 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148856 4469 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148859 4469 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:56:37.154928 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148863 4469 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148866 4469 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148871 4469 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148874 4469 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148877 4469 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148880 4469 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148883 4469 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148886 4469 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148889 4469 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148893 4469 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148895 4469 flags.go:64] FLAG: --feature-gates="" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148900 4469 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148903 4469 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148906 4469 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148909 4469 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148919 4469 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148922 4469 flags.go:64] FLAG: --help="false" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148925 4469 flags.go:64] FLAG: --hostname-override="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148928 4469 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148931 4469 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148934 4469 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148938 4469 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148941 4469 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:56:37.155553 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148945 4469 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148948 4469 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148951 4469 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148954 4469 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148957 4469 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148960 4469 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148964 4469 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148967 4469 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148970 4469 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148973 4469 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148976 4469 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148980 4469 flags.go:64] FLAG: --lock-file="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148983 4469 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148986 4469 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148989 4469 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148995 4469 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.148998 4469 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149001 4469 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149004 4469 flags.go:64] FLAG: --logging-format="text" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149007 4469 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149011 4469 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149013 4469 flags.go:64] FLAG: --manifest-url="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149016 4469 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149021 4469 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149024 4469 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:56:37.156202 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149034 4469 flags.go:64] FLAG: --max-pods="110" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149037 4469 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149040 4469 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149043 4469 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149046 4469 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149049 4469 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149052 4469 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149056 4469 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149063 4469 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149066 4469 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149069 4469 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149072 4469 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149075 4469 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149078 4469 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149081 4469 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149084 4469 flags.go:64] FLAG: --port="10250" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149088 4469 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149090 4469 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01b0ea0a7f519b1e0" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149095 4469 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149098 4469 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149102 4469 flags.go:64] FLAG: --register-node="true" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149104 4469 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149108 4469 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149111 4469 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149114 4469 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149117 4469 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:56:37.156822 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149121 4469 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149125 4469 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149128 4469 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149131 4469 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149134 4469 flags.go:64] FLAG: --runonce="false" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149137 4469 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149140 4469 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149149 4469 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149152 4469 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149155 4469 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149158 4469 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149161 4469 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149165 4469 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149169 4469 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149172 4469 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149175 4469 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149178 4469 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149181 4469 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149184 4469 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149187 4469 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149192 4469 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149195 4469 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149197 4469 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149204 4469 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149208 4469 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:56:37.157891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149211 4469 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149214 4469 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149217 4469 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149220 4469 flags.go:64] FLAG: --v="2" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149225 4469 flags.go:64] FLAG: --version="false" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149229 4469 flags.go:64] FLAG: --vmodule="" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149233 4469 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149236 4469 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149414 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149419 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149422 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149425 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149428 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149431 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149433 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149443 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149446 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149449 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149452 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149454 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149458 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.158577 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149461 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149464 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149466 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149469 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149472 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149475 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149477 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149480 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149483 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149485 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149490 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149492 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149495 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149498 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149500 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149503 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149507 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149510 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149514 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.159117 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149517 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149520 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149523 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149526 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149528 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149531 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149533 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149536 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149544 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149547 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149550 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149552 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149559 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149561 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149564 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149567 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149569 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149572 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149575 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149577 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.159571 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149580 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149582 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149585 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149589 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149592 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149594 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149597 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149600 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149602 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149605 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149607 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149610 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149612 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149615 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149618 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149620 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149623 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149626 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149628 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149631 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.160120 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149633 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149642 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149645 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149647 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149651 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149654 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149656 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149659 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149663 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149667 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149670 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149679 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149682 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149685 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149688 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149692 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149694 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149697 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149700 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.160608 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149702 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149705 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149707 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149710 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149713 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149716 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149718 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149721 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149724 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149726 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149729 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149732 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149734 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149737 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149739 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.161086 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.149766 4469 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149848 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149853 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149858 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149861 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149864 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149867 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149869 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149872 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149875 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149877 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149880 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149882 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149887 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149889 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149892 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149895 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149897 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149900 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.161442 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149903 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149906 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149909 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149912 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149914 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149917 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149919 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149922 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149925 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149927 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149930 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149933 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149935 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149938 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149941 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149944 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149948 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149951 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149954 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149958 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.161906 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149962 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149965 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149968 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149970 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149973 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149976 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149980 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149983 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149986 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149988 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149991 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149994 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149996 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.149999 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150001 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150004 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150007 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150009 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150012 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.162384 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150015 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150018 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150020 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150023 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150025 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150028 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150030 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150033 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150036 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150040 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150043 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150047 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150049 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150052 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150055 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150057 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150060 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150063 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150066 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150069 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.162854 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150072 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150075 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150077 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150080 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150082 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150085 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150088 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150091 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150094 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150096 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150099 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150102 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150104 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150107 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150109 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150113 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150117 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150120 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150122 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.163448 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150126 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150129 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150134 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150136 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150139 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150142 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150145 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150147 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150150 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.150152 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.153700 4469 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.153714 4469 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.153733 4469 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.153738 4469 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153792 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153812 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153817 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153821 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153824 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153827 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.163942 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153830 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153833 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153835 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153838 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153841 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153844 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153846 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153849 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153851 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153854 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153857 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153860 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153862 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153865 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153870 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153872 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153875 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153878 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153880 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.164426 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153884 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153887 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153890 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153893 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153895 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153898 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153901 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153903 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153906 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153908 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153912 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153916 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153919 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153923 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153925 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153928 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153931 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153933 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153936 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.164970 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153939 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153941 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153944 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153946 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153949 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153952 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153955 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153958 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153960 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153963 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153967 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153969 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153972 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153975 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153978 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153981 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153983 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153986 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153989 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153991 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.165418 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153994 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153996 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.153999 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154002 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154004 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154007 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154009 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154012 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154015 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154017 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154020 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154022 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154026 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154030 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154032 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154035 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154038 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154041 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154043 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.165915 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154046 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154048 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154051 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154054 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154057 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154059 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154062 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154065 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154068 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154070 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154073 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154076 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154079 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154081 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154084 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154087 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154089 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154092 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154094 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154097 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.166371 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154100 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154102 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154105 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.154130 4469 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154262 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154267 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154271 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154274 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154277 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154279 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154282 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154285 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154288 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154290 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154293 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154296 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.166860 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154299 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154302 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154305 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154308 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154311 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154314 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154316 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154319 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154321 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154324 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154326 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154329 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154332 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154334 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154337 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154339 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154342 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154344 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154347 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154350 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.167256 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154353 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154355 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154358 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154361 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154363 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154366 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154369 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154371 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154374 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154377 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154379 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154382 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154384 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154387 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154391 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154393 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154396 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154400 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154403 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.167759 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154406 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154410 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154413 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154415 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154418 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154421 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154424 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154426 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154429 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154431 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154434 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154436 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154439 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154441 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154444 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154447 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154451 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154455 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154458 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.168233 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154461 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154464 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154466 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154469 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154471 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154474 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154477 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154479 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154482 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154485 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154488 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154490 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154493 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154495 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154498 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154500 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154503 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154506 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154508 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154511 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.168677 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154513 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154516 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154519 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154521 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154524 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154527 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154529 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154532 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154534 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154537 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154539 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154542 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154544 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154547 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154553 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154555 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.169215 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.154578 4469 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154609 4469 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154613 4469 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154616 4469 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154621 4469 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154625 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154628 4469 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154631 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154634 4469 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154636 4469 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154639 4469 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154642 4469 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154644 4469 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154647 4469 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154650 4469 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154652 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154655 4469 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154658 4469 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154660 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:37.169607 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154663 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154666 4469 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154668 4469 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154671 4469 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154675 4469 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154679 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154682 4469 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154685 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154688 4469 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154690 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154693 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154695 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154700 4469 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154703 4469 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154706 4469 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154708 4469 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154711 4469 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154714 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154716 4469 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:37.170136 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154719 4469 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154722 4469 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154724 4469 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154728 4469 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154732 4469 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154735 4469 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154738 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154741 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154743 4469 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154746 4469 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154749 4469 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154751 4469 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154754 4469 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154756 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154759 4469 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154762 4469 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154764 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154767 4469 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154769 4469 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:37.170619 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154772 4469 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154775 4469 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154777 4469 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154780 4469 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154783 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154785 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154789 4469 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154792 4469 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154794 4469 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154812 4469 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154817 4469 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154821 4469 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154824 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154826 4469 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154830 4469 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154833 4469 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154836 4469 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154838 4469 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154841 4469 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154844 4469 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:37.171091 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154846 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154849 4469 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154851 4469 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154854 4469 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154857 4469 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154859 4469 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154862 4469 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154865 4469 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154868 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154871 4469 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154874 4469 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154876 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154879 4469 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154881 4469 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154884 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154887 4469 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154889 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154892 4469 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154898 4469 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154902 4469 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:37.171569 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154904 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154907 4469 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154910 4469 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154912 4469 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154915 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154917 4469 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154920 4469 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154923 4469 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154927 4469 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:37.154930 4469 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.154977 4469 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.155922 4469 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.155977 4469 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.156152 4469 server.go:1014] "Starting client certificate rotation" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.156298 4469 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.156348 4469 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:56 +0000 UTC" deadline="2028-05-12 19:41:39.433162769 +0000 UTC" Nov 02 14:56:37.172060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.156383 4469 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13372h45m2.276781339s" Nov 02 14:56:37.172455 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.156586 4469 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:56:37.173068 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.173048 4469 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:56:37.174459 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.174441 4469 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:56:37.176031 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.176014 4469 log.go:25] "Validated CRI v1 image API" Nov 02 14:56:37.176766 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.176747 4469 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:56:37.178276 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.178251 4469 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a28a6853-da96-4013-8309-32ad9e1f2f33:/dev/nvme0n1p3 c7983892-a0a0-49a4-a892-db41c1bb7529:/dev/nvme0n1p4] Nov 02 14:56:37.178437 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.178276 4469 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11fa4f46fad1439ecdeb0044c7282a9970a12d1e2558729678d132909b6a07d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11fa4f46fad1439ecdeb0044c7282a9970a12d1e2558729678d132909b6a07d2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f6d84dc13e9f38c205020d35bdfe16b1b8601fad82731275b29544ea85f1244/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f6d84dc13e9f38c205020d35bdfe16b1b8601fad82731275b29544ea85f1244/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9da91326d1f7a53509cf2745cc1513c4677894ca82b893902ad0ac19230df37b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9da91326d1f7a53509cf2745cc1513c4677894ca82b893902ad0ac19230df37b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8e526a2cb011b53caf302e966730dc2a9b8f35455c911ed979f690ac4b1690d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8e526a2cb011b53caf302e966730dc2a9b8f35455c911ed979f690ac4b1690d/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ba3151-5436-402c-87c6-47dbf8fbb3ce/volumes/kubernetes.io~projected/kube-api-access-pxbgl:{mountpoint:/var/lib/kubelet/pods/02ba3151-5436-402c-87c6-47dbf8fbb3ce/volumes/kubernetes.io~projected/kube-api-access-pxbgl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09413fd8-4cfb-4e13-adb8-c8a58533ad5e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/09413fd8-4cfb-4e13-adb8-c8a58533ad5e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0959e584-488f-4332-a378-b927ac207e49/volumes/kubernetes.io~projected/kube-api-access-6zwsp:{mountpoint:/var/lib/kubelet/pods/0959e584-488f-4332-a378-b927ac207e49/volumes/kubernetes.io~projected/kube-api-access-6zwsp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cdb8471-d93e-4750-9aef-984d1a738d4a/volumes/kubernetes.io~projected/kube-api-access-6l5xs:{mountpoint:/var/lib/kubelet/pods/2cdb8471-d93e-4750-9aef-984d1a738d4a/volumes/kubernetes.io~projected/kube-api-access-6l5xs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~projected/kube-api-access-hd2bz:{mountpoint:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~projected/kube-api-access-hd2bz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50dc24f3-a26e-4583-bfe3-f1a25e97315a/volumes/kubernetes.io~projected/kube-api-access-ds9pd:{mountpoint:/var/lib/kubelet/pods/50dc24f3-a26e-4583-bfe3-f1a25e97315a/volumes/kubernetes.io~projected/kube-api-access-ds9pd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51dfa937-57ef-4edc-b10d-2d995c93e764/volumes/kubernetes.io~projected/kube-api-access-kp62d:{mountpoint:/var/lib/kubelet/pods/51dfa937-57ef-4edc-b10d-2d995c93e764/volumes/kubernetes.io~projected/kube-api-access-kp62d major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8d1f65-2c06-4241-a3b4-025039c6e9e1/volumes/kubernetes.io~projected/kube-api-access-4rj9g:{mountpoint:/var/lib/kubelet/pods/5b8d1f65-2c06-4241-a3b4-025039c6e9e1/volumes/kubernetes.io~projected/kube-api-access-4rj9g major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/683f8569-4473-4d34-ae00-ba1bf3bf9f5d/volumes/kubernetes.io~projected/kube-api-access-5dsqb:{mountpoint:/var/lib/kubelet/pods/683f8569-4473-4d34-ae00-ba1bf3bf9f5d/volumes/kubernetes.io~projected/kube-api-access-5dsqb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fc4fca4-e664-4419-9ad8-a6930c959473/volumes/kubernetes.io~projected/kube-api-access-lc7x7:{mountpoint:/var/lib/kubelet/pods/6fc4fca4-e664-4419-9ad8-a6930c959473/volumes/kubernetes.io~projected/kube-api-access-lc7x7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7734be1e-a6d2-4c25-9982-3f0fbe90c330/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7734be1e-a6d2-4c25-9982-3f0fbe90c330/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e14b273-2cc6-4872-88e3-fec55e2b0631/volumes/kubernetes.io~projected/kube-api-access-glcf9:{mountpoint:/var/lib/kubelet/pods/8e14b273-2cc6-4872-88e3-fec55e2b0631/volumes/kubernetes.io~projected/kube-api-access-glcf9 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~projected/kube-api-access-vkq7b:{mountpoint:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~projected/kube-api-access-vkq7b major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~projected/kube-api-access-c256h:{mountpoint:/var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~projected/kube-api-access-c256h major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab3ac100-fe63-4c64-8587-6c137b20f5d5/volumes/kubernetes.io~projected/kube-api-access-k7lvk:{mountpoint:/var/lib/kubelet/pods/ab3ac100-fe63-4c64-8587-6c137b20f5d5/volumes/kubernetes.io~projected/kube-api-access-k7lvk major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0bde169-ce7b-431d-a75e-fc6c56fbe779/volumes/kubernetes.io~projected/kube-api-access-q6l28:{mountpoint:/var/lib/kubelet/pods/f0bde169-ce7b-431d-a75e-fc6c56fbe779/volumes/kubernetes.io~projected/kube-api-access-q6l28 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f473c9a3-09d1-407c-be89-b00857fd2f56/volumes/kubernetes.io~projected/kube-api-access-w65c8:{mountpoint:/var/lib/kubelet/pods/f473c9a3-09d1-407c-be89-b00857fd2f56/volumes/kubernetes.io~projected/kube-api-access-w65c8 major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6baea9658b80614f4e3a977bd161689a441f426bfc5c923485d55c7b84af492a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c451f0d18d44ad229d89dfecc068f5e74d8f7fdb56d3f4a6e0f96dd003081ad1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9c7102e38e80822c79c09117587f1ee4a655513933b048162516b193db80758c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/13201b2eaca98b028b3bd9718270780e1b05139274867a4d8c4022928b4070b6/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/207ade7cabe2601d9900c2e1394c03f8d7f8befb3c94ea6914d41b22145e61a2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/760aa8c4d6a6dcedca4ea15ab9d18ed5bff0d72c2a44dba05b8bb58cf463b0c4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2d574f7500cbf544deaea4c7b65f7d9f6c4e81cfbe35702965a7ca10cd24d7bb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6749dc94071c9ba2f5bf606f3c0b45fbbe5389dfe7477327ce7f5e8dc303c3e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a129cb761b776bcf7f414b883d61e03c9bbb3996cc92d1f353717db9d0ee7833/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/54ea752d6c099747bd069c25c5235794911e37fbcdd25f8f59676f169aa84384/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e876f948eee2970e5e280c32bae21253ac9ca9f4fc58ba2a083ea3e4f60753f6/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/ea9f8c87946a49b06ae18d9c6cf9f104f7ac988e18b51c81f5b0b55d2df83269/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9bed8b0863baa1cf88ec456e1a2093ea624f6546b88abd2d7d1994e5b2c1548a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/371ae6f3f2c25b6108970bfcdf1911fe750e5dedb0841acba17d8beea21cdf1a/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/9f862b7629e17d0291e4ec556e353e9b141a3935c8790667f0b96487c97bc3e0/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/d95e8bf7610df038ac6705104d0e78f192b0d75e4380fab3504754114cf32ada/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e7dd22e39dd12adf055270e1a4e0954ad467afd6de467efec071e1229578fb9a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/68a0ed6e7d26e3bc9fdd58ca1350059227b11c160635facb0dd76b481a3bf0f5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c5c013a8feb6f0117c503e21977fa16f53447c91eef9e5177cc5deb052d56605/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f746bf686fa22a95238cadf8becd0d74e95b6c9e2d489b5c59569bbe50898c99/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/700078a0995b764692bf03c312d29a9dcc98aeebe7c7228fa400e4f5829a69d3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/9cee7e4a21af71d360324dec2b6b13ba8343aa6e53756003b55df60424f01393/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c72c62d1c9f36bbf4e3f944a2fc615a723ee1635a0db26480b45bafd51cdce89/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1d3218084e6fdb4f55c014604f1bd2c5e7b7065d03b7738d75abf994d869e42d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/ecad98a9f4c564b43f34bd5703da4e36a73d4b902c03403c17606a02d170948a/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/267a0d0e5e9419f12f404c651560b4fec71072517e7169ec0fe5cb756e4d56eb/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/428d149645ceba18c094003c17636317f6fac54b203aaa7ff0e882a8a8ffd297/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/62793ce4ffccedabfaf3f5327530ca5f5a36c0bfd9532d9177bc38067383aa12/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/339d482c4e0d559d45c26768fbf1d11a6098faf673b1ece964001e54af3ad371/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/6d581eda5d6e40ad9f0640062c35899057e5629fc316d04e8316c47977f16b31/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/b0bd08bee2e441202ba628f60027062e94d0e58333c338d4c03e04628638801a/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fc99acbee42f6e58a02ea589edc94a362b3eeee9657386e7200f35c2e2b9b49d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f2f04e85351a09e64eb3e45628450d5d752a3368908c99a0b376dd2d443de745/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ae42f303dc5190edb10f22d0290289f43d381d0f4f04dc94aa8f588a09ed3d24/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c23781bb50d2fc0ff3a6977a84a790d1c06eaf06ff63fdf81a761f3a019ac783/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e988a2ec2be5367d11daa88c5a38702090d1d82bc02d38b14a13e5b833fd02fc/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b035c316e5e6da806a96600cae7209d9e9d77e93ca64d60ca8e2b3122a733d67/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0c6a760ecaa0e44f51071442fe738067409303201323b73d3e0d961b29ce761c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bea0543eaec7ae821c37c8bcf79849da52191300a21cd880b6ee90626815151a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fd75c38d60939c3dc7d48a036118bf522e7ed5c91bf26eec7d6e52e5ed076744/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4844d4b646930a1f88e952485746db9ffc9a0e210543d07271177909c319973b/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 14:56:37.185083 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.184832 4469 manager.go:217] Machine: {Timestamp:2026-11-02 14:56:37.184356739 +0000 UTC m=+0.067959249 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082304 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27cc97ddbb408310eef35eb35480d7 SystemUUID:ec27cc97-ddbb-4083-10ee-f35eb35480d7 BootID:08b01d73-5f3b-43d3-a941-8889b7ebca23 Filesystems:[{Device:/run/containers/storage/overlay-containers/ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51dfa937-57ef-4edc-b10d-2d995c93e764/volumes/kubernetes.io~projected/kube-api-access-kp62d DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8d1f65-2c06-4241-a3b4-025039c6e9e1/volumes/kubernetes.io~projected/kube-api-access-4rj9g DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9da91326d1f7a53509cf2745cc1513c4677894ca82b893902ad0ac19230df37b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e14b273-2cc6-4872-88e3-fec55e2b0631/volumes/kubernetes.io~projected/kube-api-access-glcf9 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a13f26396a50cccbff972e71bb71aec27eef5ea57e1e6c427695dfac4b218d01/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31ec8e29f40ff1bc4ff1057e8737119a6ff8cf4180f41875b55ef04f805afcad/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a576234597d44fb89ade76a35b70b10b0f7a81e892bb5b5eec8c57d991691f30/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbbc707b5f33b0f7cf9bb0dd0519e6f5f0e35fc56724b74c360256541b7ba59c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cdb8471-d93e-4750-9aef-984d1a738d4a/volumes/kubernetes.io~projected/kube-api-access-6l5xs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0bde169-ce7b-431d-a75e-fc6c56fbe779/volumes/kubernetes.io~projected/kube-api-access-q6l28 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0959e584-488f-4332-a378-b927ac207e49/volumes/kubernetes.io~projected/kube-api-access-6zwsp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~projected/kube-api-access-hd2bz DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11fa4f46fad1439ecdeb0044c7282a9970a12d1e2558729678d132909b6a07d2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7734be1e-a6d2-4c25-9982-3f0fbe90c330/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9ff2024d058decf61fbe331c31094d28d69dc1f6c03e694048e3549a47ec315/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52655513c16d598e5ed71cb5d8eb929de4583e050f91ba17d834079de23330b0/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f9f41550bc8ac48a2bf831ef4eaca735a10b0ad24a373da983cdbfa88ae3a81/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09413fd8-4cfb-4e13-adb8-c8a58533ad5e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6fc4fca4-e664-4419-9ad8-a6930c959473/volumes/kubernetes.io~projected/kube-api-access-lc7x7 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f473c9a3-09d1-407c-be89-b00857fd2f56/volumes/kubernetes.io~projected/kube-api-access-w65c8 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a51a2287-9459-4c81-8646-15515c5439d9/volumes/kubernetes.io~projected/kube-api-access-c256h DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/683f8569-4473-4d34-ae00-ba1bf3bf9f5d/volumes/kubernetes.io~projected/kube-api-access-5dsqb DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46626844-9af4-499e-8f53-7428a1deb452/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab3ac100-fe63-4c64-8587-6c137b20f5d5/volumes/kubernetes.io~projected/kube-api-access-k7lvk DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50dc24f3-a26e-4583-bfe3-f1a25e97315a/volumes/kubernetes.io~projected/kube-api-access-ds9pd DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32eff3c4e31c50e71cdd1b3ebbb782884995712b317e9c788ee9c26d0d743e19/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91058aad-5292-4b62-ab2e-91444f80c847/volumes/kubernetes.io~projected/kube-api-access-vkq7b DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02ba3151-5436-402c-87c6-47dbf8fbb3ce/volumes/kubernetes.io~projected/kube-api-access-pxbgl DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8e526a2cb011b53caf302e966730dc2a9b8f35455c911ed979f690ac4b1690d/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f6d84dc13e9f38c205020d35bdfe16b1b8601fad82731275b29544ea85f1244/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f9f41550bc8ac4 MacAddress:16:52:98:7c:4c:0c Speed:10000 Mtu:8901} {Name:52655513c16d598 MacAddress:c2:bb:31:e2:46:5e Speed:10000 Mtu:8901} {Name:a13f26396a50ccc MacAddress:a6:64:f9:fb:fa:04 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:13:27:d4:85:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:8f:4a:d7:4a:0d Speed:0 Mtu:8901} {Name:d5ec43199f1657b MacAddress:aa:49:d9:19:6d:cd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:13:27:d4:85:95 Speed:0 Mtu:9001} {Name:f9ff2024d058dec MacAddress:c2:4d:e9:49:27:41 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:e0:17:a0:f2:e7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:b4:a1:b9:43:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:56:37.185083 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185085 4469 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:56:37.185308 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185130 4469 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:56:37.185415 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185397 4469 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:56:37.185549 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185415 4469 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-103.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:56:37.185591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185551 4469 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:56:37.185591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185560 4469 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:56:37.185591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185573 4469 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:56:37.185591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185586 4469 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:56:37.185693 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185669 4469 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:56:37.185884 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185874 4469 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:56:37.185998 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.185988 4469 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:56:37.186034 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.186027 4469 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:56:37.186062 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.186040 4469 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:56:37.186062 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.186057 4469 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:56:37.186111 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.186066 4469 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:56:37.186111 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.186079 4469 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:56:37.188128 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.187390 4469 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:56:37.188128 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.187412 4469 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:56:37.188128 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.187982 4469 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:56:37.191139 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191118 4469 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:56:37.191139 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191138 4469 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:56:37.191268 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191255 4469 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 14:56:37.191534 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191520 4469 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:56:37.191588 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191567 4469 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:56:37.191704 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191693 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:56:37.191733 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191713 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:56:37.191733 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191723 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:56:37.191733 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191732 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191742 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191751 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191761 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191769 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191779 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191787 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191817 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:56:37.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191832 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:56:37.192079 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191863 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:56:37.192079 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.191873 4469 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:56:37.195233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.195218 4469 server.go:1263] "Started kubelet" Nov 02 14:56:37.195591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.195280 4469 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:56:37.195591 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.195315 4469 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:56:37.195939 ip-10-0-135-103 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:56:37.196859 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.195954 4469 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:56:37.196859 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.196396 4469 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:56:37.196859 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.196521 4469 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:56:37.198106 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198090 4469 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198113 4469 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198167 4469 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198177 4469 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198164 4469 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:58 +0000 UTC" deadline="2028-08-20 02:54:36.934051614 +0000 UTC" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198195 4469 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15755h57m59.735859655s" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.198238 4469 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.198249 4469 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199094 4469 factory.go:55] Registering systemd factory Nov 02 14:56:37.199129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199109 4469 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:56:37.199862 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199833 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a51a2287-9459-4c81-8646-15515c5439d9" volumeName="kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert" seLinuxMountContext="" Nov 02 14:56:37.199905 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199868 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46626844-9af4-499e-8f53-7428a1deb452" volumeName="kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config" seLinuxMountContext="" Nov 02 14:56:37.199905 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199885 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8d1f65-2c06-4241-a3b4-025039c6e9e1" volumeName="kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy" seLinuxMountContext="" Nov 02 14:56:37.199905 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199898 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a51a2287-9459-4c81-8646-15515c5439d9" volumeName="kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199910 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bde169-ce7b-431d-a75e-fc6c56fbe779" volumeName="kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199921 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46626844-9af4-499e-8f53-7428a1deb452" volumeName="kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199934 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dfa937-57ef-4edc-b10d-2d995c93e764" volumeName="kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199948 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="683f8569-4473-4d34-ae00-ba1bf3bf9f5d" volumeName="kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199963 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7734be1e-a6d2-4c25-9982-3f0fbe90c330" volumeName="kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199974 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e14b273-2cc6-4872-88e3-fec55e2b0631" volumeName="kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199986 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e14b273-2cc6-4872-88e3-fec55e2b0631" volumeName="kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path" seLinuxMountContext="" Nov 02 14:56:37.200016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.199998 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bde169-ce7b-431d-a75e-fc6c56fbe779" volumeName="kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200010 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f473c9a3-09d1-407c-be89-b00857fd2f56" volumeName="kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200047 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ba3151-5436-402c-87c6-47dbf8fbb3ce" volumeName="kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200064 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46626844-9af4-499e-8f53-7428a1deb452" volumeName="kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200075 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8d1f65-2c06-4241-a3b4-025039c6e9e1" volumeName="kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200087 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fc4fca4-e664-4419-9ad8-a6930c959473" volumeName="kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200097 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91058aad-5292-4b62-ab2e-91444f80c847" volumeName="kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200108 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f473c9a3-09d1-407c-be89-b00857fd2f56" volumeName="kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200118 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0959e584-488f-4332-a378-b927ac207e49" volumeName="kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200128 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46626844-9af4-499e-8f53-7428a1deb452" volumeName="kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200139 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f473c9a3-09d1-407c-be89-b00857fd2f56" volumeName="kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200150 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0959e584-488f-4332-a378-b927ac207e49" volumeName="kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200177 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cdb8471-d93e-4750-9aef-984d1a738d4a" volumeName="kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200189 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dfa937-57ef-4edc-b10d-2d995c93e764" volumeName="kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200203 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dfa937-57ef-4edc-b10d-2d995c93e764" volumeName="kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200216 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fc4fca4-e664-4419-9ad8-a6930c959473" volumeName="kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200228 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91058aad-5292-4b62-ab2e-91444f80c847" volumeName="kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200247 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab3ac100-fe63-4c64-8587-6c137b20f5d5" volumeName="kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200260 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09413fd8-4cfb-4e13-adb8-c8a58533ad5e" volumeName="kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200272 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cdb8471-d93e-4750-9aef-984d1a738d4a" volumeName="kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200284 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50dc24f3-a26e-4583-bfe3-f1a25e97315a" volumeName="kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd" seLinuxMountContext="" Nov 02 14:56:37.200291 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200296 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dfa937-57ef-4edc-b10d-2d995c93e764" volumeName="kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200308 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7734be1e-a6d2-4c25-9982-3f0fbe90c330" volumeName="kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200320 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91058aad-5292-4b62-ab2e-91444f80c847" volumeName="kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200334 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46626844-9af4-499e-8f53-7428a1deb452" volumeName="kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200345 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8d1f65-2c06-4241-a3b4-025039c6e9e1" volumeName="kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200357 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e14b273-2cc6-4872-88e3-fec55e2b0631" volumeName="kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200369 4469 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e14b273-2cc6-4872-88e3-fec55e2b0631" volumeName="kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9" seLinuxMountContext="" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200381 4469 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:56:37.200921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.200388 4469 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:56:37.204849 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.204827 4469 factory.go:153] Registering CRI-O factory Nov 02 14:56:37.204849 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.204851 4469 factory.go:223] Registration of the crio container factory successfully Nov 02 14:56:37.205042 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.204940 4469 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:56:37.205042 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.204961 4469 factory.go:103] Registering Raw factory Nov 02 14:56:37.205042 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.204979 4469 manager.go:1199] Started watching for new ooms in manager Nov 02 14:56:37.205322 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.205309 4469 manager.go:319] Starting recovery of all containers Nov 02 14:56:37.229404 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.229361 4469 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:56:37.230867 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.230845 4469 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:56:37.230867 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.230865 4469 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:56:37.231014 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.230883 4469 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:56:37.231014 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.230915 4469 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 14:56:37.251598 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.251563 4469 generic.go:356] "Generic (PLEG): container finished" podID="aea26f2febf8f66e888f887686e2c7f4" containerID="39e2ea0aa2c18cdc40da410f042f02f4a1060d0d4e21816c409d058ceb59bd81" exitCode=0 Nov 02 14:56:37.257762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257742 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="4718ed82e63c8de270a4bf0ebfdb6097c3797dbe492fd652250cf7579ea098d0" exitCode=0 Nov 02 14:56:37.257762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257759 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="3398679e405883a7f2ecfa8c556d2ef06c217bb2586c675c50888c77b662bcce" exitCode=0 Nov 02 14:56:37.257932 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257767 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="8fcbeaef1a14ece725300cdd6ff385e24c8d40e93a1e931f3409717ab47b48a6" exitCode=0 Nov 02 14:56:37.257932 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257774 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="53f0b142c38efcac971a5f9d64544bfc6cb6b15e910be3e3ceb169114c3b18cc" exitCode=0 Nov 02 14:56:37.257932 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257781 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="c0a97e687d9501b585b036dc8b4df7dc97b16c5f919cc1181864708c1f1fe30f" exitCode=0 Nov 02 14:56:37.257932 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.257787 4469 generic.go:356] "Generic (PLEG): container finished" podID="51dfa937-57ef-4edc-b10d-2d995c93e764" containerID="8a4889e4786c8d5eae8eef0c4adb0d133762100371b06aec6a8e8433122dec58" exitCode=0 Nov 02 14:56:37.272661 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.272643 4469 manager.go:324] Recovery completed Nov 02 14:56:37.284924 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.284908 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.286115 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.286100 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.286168 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.286128 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.286168 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.286138 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.287485 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287471 4469 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:56:37.287485 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287482 4469 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:56:37.287561 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287489 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.287618 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287498 4469 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:56:37.287692 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287679 4469 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 14:56:37.287726 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287692 4469 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 14:56:37.287726 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287709 4469 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 14:56:37.287726 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287714 4469 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 14:56:37.287726 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287721 4469 policy_none.go:50] "Start" Nov 02 14:56:37.287864 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287728 4469 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:56:37.287864 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.287738 4469 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:56:37.288469 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.288455 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.288507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.288482 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.288507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.288497 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.296019 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.296004 4469 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 14:56:37.296077 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.296021 4469 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 14:56:37.296077 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.296029 4469 policy_none.go:44] "Start" Nov 02 14:56:37.298477 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.298463 4469 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:56:37.299955 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.299929 4469 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:56:37.302563 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.302546 4469 manager.go:341] "Starting Device Plugin manager" Nov 02 14:56:37.302634 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.302576 4469 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:56:37.302634 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.302588 4469 server.go:87] "Starting device plugin registration server" Nov 02 14:56:37.302909 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.302897 4469 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:56:37.302947 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.302912 4469 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:56:37.303627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.303013 4469 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:56:37.303627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.303135 4469 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:56:37.303627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.303175 4469 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:56:37.303627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.303181 4469 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:56:37.303627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.303240 4469 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:37.303163255Z","UUID":"c4666c32-44a9-45a7-9318-e55960ff0209","Handler":null,"Name":"","Endpoint":""} Nov 02 14:56:37.304454 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.304435 4469 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:56:37.304585 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.304472 4469 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" Nov 02 14:56:37.304765 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.304747 4469 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:56:37.304892 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.304771 4469 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:56:37.331948 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.331321 4469 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal"] Nov 02 14:56:37.332041 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.331996 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.333016 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.332995 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.333100 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.333027 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.333100 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.333038 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.333274 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.333264 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.333421 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.333407 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.334137 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334120 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.334137 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334131 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.334233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334149 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.334233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334153 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.334233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334162 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.334233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334163 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.334354 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.334290 4469 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.334468 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334441 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerStarted","Data":"d14cfb9e8750143fafe53134fdb474c8592e52e621e4acaaf552c57b36b31c06"} Nov 02 14:56:37.334502 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334480 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerDied","Data":"39e2ea0aa2c18cdc40da410f042f02f4a1060d0d4e21816c409d058ceb59bd81"} Nov 02 14:56:37.334502 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334496 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"aea26f2febf8f66e888f887686e2c7f4","Type":"ContainerStarted","Data":"1285df7aa17abe2ef348edc0bbfc7ce326b81c5caedead9ea758f31967a93dcb"} Nov 02 14:56:37.334554 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334530 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"5f06372767f35a0b77d37ba2e829e38f","Type":"ContainerStarted","Data":"a5dbdd3512bb06d5f7df705bd12f6afc533a56cc6563b28ff5115a689ede80e4"} Nov 02 14:56:37.334554 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334548 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" event={"ID":"5f06372767f35a0b77d37ba2e829e38f","Type":"ContainerStarted","Data":"ea0186b94395ed42df7c33c4105cd98ae1decbad7147b710e2dc35293a5f4ff8"} Nov 02 14:56:37.334621 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.334610 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.335485 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.335470 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.335559 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.335489 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.335559 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.335498 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.335619 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.335612 4469 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.us-west-1.compute.internal\" not found" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.395387 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.395356 4469 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:37.396270 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.396239 4469 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:37.399426 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.399409 4469 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.403559 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.403546 4469 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:37.404648 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.404632 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:37.404734 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.404664 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:37.404734 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.404679 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:37.404734 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.404705 4469 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.446008 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.445981 4469 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:37.446222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.446207 4469 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:37.446351 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.446336 4469 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 14:56:37.462081 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.462061 4469 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:37.462157 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.462109 4469 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.462157 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.462121 4469 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.520761 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.520736 4469 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:37.520884 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:37.520779 4469 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.546979 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.546954 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.547063 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.546985 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.547063 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.547050 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.548863 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.548849 4469 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.548917 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.548908 4469 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.549457 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.549443 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeNotReady" Nov 02 14:56:37.549515 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.549471 4469 setters.go:546] "Node became not ready" node="ip-10-0-135-103.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T14:56:37Z","lastTransitionTime":"2026-11-02T14:56:37Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 14:56:37.559973 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.559952 4469 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-103.us-west-1.compute.internal" is owned by different node (oldNodeID="06d70dec-7d97-4d7b-9f28-b533f17aa001", newNodeID=""), cleaning up... Nov 02 14:56:37.647850 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647757 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.647850 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647796 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.647850 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647838 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.648040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647866 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.648040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647882 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5f06372767f35a0b77d37ba2e829e38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"5f06372767f35a0b77d37ba2e829e38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.648040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.647934 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea26f2febf8f66e888f887686e2c7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" (UID: \"aea26f2febf8f66e888f887686e2c7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:37.819982 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.819952 4469 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-103.us-west-1.compute.internal" event="NodeReady" Nov 02 14:56:37.820136 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:37.820040 4469 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 14:56:38.187131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.187099 4469 apiserver.go:52] "Watching apiserver" Nov 02 14:56:38.267523 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.267495 4469 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:38.267661 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.267567 4469 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:38.338414 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.338386 4469 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:38.338582 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.338436 4469 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:38.338582 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.338386 4469 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:38.338582 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.338493 4469 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal" Nov 02 14:56:38.338965 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.338605 4469 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:56:38.339614 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.339578 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7x9dz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92","openshift-network-diagnostics/network-check-target-wx98q","kube-system/global-pull-secret-syncer-tzjrs","kube-system/konnectivity-agent-lw49d","openshift-multus/multus-additional-cni-plugins-5lqlt","openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc","openshift-ingress-canary/ingress-canary-vxl2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-gffg7","openshift-insights/periodic-gathering-tgkg9-9n6bd","openshift-network-operator/iptables-alerter-b5jgk","openshift-multus/multus-65hmq","openshift-multus/network-metrics-daemon-245v8","kube-system/kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal","openshift-image-registry/node-ca-bzr9k","openshift-insights/insights-runtime-extractor-4prgg","openshift-ovn-kubernetes/ovnkube-node-wpqbc","openshift-dns/dns-default-w7g64"] Nov 02 14:56:38.342910 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.342664 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:38.345948 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.345931 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.346735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.346719 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.399706 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.399675 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:56:38.399964 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.399948 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:56:38.400057 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400037 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dn6m\"" Nov 02 14:56:38.400125 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400037 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zct6j\"" Nov 02 14:56:38.400213 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400198 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dh847\"" Nov 02 14:56:38.400417 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400400 4469 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:56:38.400494 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400477 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.400547 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400504 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:56:38.400547 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400516 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zp9vx\"" Nov 02 14:56:38.400647 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400614 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:56:38.400647 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400625 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snsjj\"" Nov 02 14:56:38.400738 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400671 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrdxh\"" Nov 02 14:56:38.400738 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400718 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5w9z9\"" Nov 02 14:56:38.400843 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400748 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:56:38.400843 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400751 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.400949 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400933 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:56:38.400997 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400972 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:56:38.400997 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.400989 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:56:38.401069 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401043 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:56:38.401115 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401070 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401115 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401089 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401115 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401088 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:56:38.401225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401128 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.401225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401132 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.401225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401152 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401207 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401334 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401229 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401334 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401235 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401334 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401258 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401334 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401258 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.401498 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401390 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.401498 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401480 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nw765\"" Nov 02 14:56:38.401498 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401488 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:56:38.401498 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401499 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.401707 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401510 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ptvxg\"" Nov 02 14:56:38.401707 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401540 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.401707 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401574 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:56:38.401707 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401603 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:56:38.401707 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401629 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:56:38.401978 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.401963 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5lvgl\"" Nov 02 14:56:38.402038 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.402025 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4lkk\"" Nov 02 14:56:38.440501 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.440451 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.443066 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.443052 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.450713 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450692 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:56:38.450793 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450747 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.450876 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450851 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fbnz4\"" Nov 02 14:56:38.450931 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450855 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.450931 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450870 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:56:38.450931 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450909 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:56:38.451068 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.450915 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 14:56:38.451068 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.451049 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dhf26\"" Nov 02 14:56:38.451728 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.451707 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.451811 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.451774 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:56:38.451895 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.451881 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mnrgm\"" Nov 02 14:56:38.452011 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.451995 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mcvdb\"" Nov 02 14:56:38.453156 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453136 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:56:38.453279 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453264 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wtdgk\"" Nov 02 14:56:38.453333 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453319 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:56:38.453404 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453391 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bm8rt\"" Nov 02 14:56:38.453463 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453425 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 14:56:38.453509 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453483 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:56:38.453555 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453517 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:56:38.453772 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453753 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.453839 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453793 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.453897 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453842 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.453897 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453878 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:38.454040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453909 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.454040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453936 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.454040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.453963 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.454169 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454051 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.454169 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454138 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwsp\" (UniqueName: \"kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.454266 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454189 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.454266 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454200 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-etc-tuned\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.454358 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454337 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.454407 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454382 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.454459 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454418 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.454459 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454444 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.454540 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454503 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.454593 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454556 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.454643 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454618 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.454691 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454643 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f473c9a3-09d1-407c-be89-b00857fd2f56-config-volume\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.454691 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454667 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.454782 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454727 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.454782 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454750 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-original-pull-secret\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.454782 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454771 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.454947 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454859 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pd\" (UniqueName: \"kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:38.454997 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.454962 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c256h\" (UniqueName: \"kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:38.455062 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455000 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.455062 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455019 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455062 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455034 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455165 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455066 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455200 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455128 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp62d\" (UniqueName: \"kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455578 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455556 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:38.455636 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455614 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.455636 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455638 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.455735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455657 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l28\" (UniqueName: \"kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.455735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455671 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dfa937-57ef-4edc-b10d-2d995c93e764-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.455735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455692 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsqb\" (UniqueName: \"kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb\") pod \"migrator-66df5d496b-l69rc\" (UID: \"683f8569-4473-4d34-ae00-ba1bf3bf9f5d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:38.455943 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455835 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.455943 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455861 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.455943 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455884 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.455943 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455910 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455952 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455961 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-daemon-config\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455957 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7734be1e-a6d2-4c25-9982-3f0fbe90c330-agent-certs\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.455983 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456083 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456108 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456127 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456147 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-script-lib\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456151 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq7b\" (UniqueName: \"kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456180 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91058aad-5292-4b62-ab2e-91444f80c847-tmp\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456183 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456204 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456222 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456250 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456314 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cni-binary-copy\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456329 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456362 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.456412 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456389 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456418 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6fc4fca4-e664-4419-9ad8-a6930c959473-iptables-alerter-script\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456421 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456461 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456488 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456492 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb8471-d93e-4750-9aef-984d1a738d4a-tmp-dir\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456507 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456517 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456524 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbgl\" (UniqueName: \"kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456540 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456556 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456573 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456586 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456610 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456617 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456679 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456715 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456753 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456819 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65c8\" (UniqueName: \"kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.456869 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456863 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456889 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a51a2287-9459-4c81-8646-15515c5439d9-cert\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456951 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.456978 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0bde169-ce7b-431d-a75e-fc6c56fbe779-data-volume\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457015 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457060 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457103 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457133 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457151 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rj9g\" (UniqueName: \"kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457164 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0959e584-488f-4332-a378-b927ac207e49-serviceca\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457194 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457217 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457232 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457264 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457284 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457295 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7734be1e-a6d2-4c25-9982-3f0fbe90c330-konnectivity-ca\") pod \"konnectivity-agent-lw49d\" (UID: \"7734be1e-a6d2-4c25-9982-3f0fbe90c330\") " pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457316 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457349 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457373 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457390 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457422 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457441 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457481 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5xs\" (UniqueName: \"kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457509 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457532 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.457537 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457551 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-env-overrides\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457580 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f473c9a3-09d1-407c-be89-b00857fd2f56-tmp-dir\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457578 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457621 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457648 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457662 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457710 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457737 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457753 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457783 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2bz\" (UniqueName: \"kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457814 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457816 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46626844-9af4-499e-8f53-7428a1deb452-ovnkube-config\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457823 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46626844-9af4-499e-8f53-7428a1deb452-ovn-node-metrics-cert\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457890 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457908 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457943 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457970 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.457995 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.458237 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.519448 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.519423 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwsp\" (UniqueName: \"kubernetes.io/projected/0959e584-488f-4332-a378-b927ac207e49-kube-api-access-6zwsp\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.521776 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.521739 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9pd\" (UniqueName: \"kubernetes.io/projected/50dc24f3-a26e-4583-bfe3-f1a25e97315a-kube-api-access-ds9pd\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:38.521856 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.521821 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lvk\" (UniqueName: \"kubernetes.io/projected/ab3ac100-fe63-4c64-8587-6c137b20f5d5-kube-api-access-k7lvk\") pod \"network-check-target-wx98q\" (UID: \"ab3ac100-fe63-4c64-8587-6c137b20f5d5\") " pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:38.526685 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.526664 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp62d\" (UniqueName: \"kubernetes.io/projected/51dfa937-57ef-4edc-b10d-2d995c93e764-kube-api-access-kp62d\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.526956 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.526939 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65c8\" (UniqueName: \"kubernetes.io/projected/f473c9a3-09d1-407c-be89-b00857fd2f56-kube-api-access-w65c8\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.527080 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.527063 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rj9g\" (UniqueName: \"kubernetes.io/projected/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-kube-api-access-4rj9g\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.527154 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.527139 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c256h\" (UniqueName: \"kubernetes.io/projected/a51a2287-9459-4c81-8646-15515c5439d9-kube-api-access-c256h\") pod \"ingress-canary-vxl2w\" (UID: \"a51a2287-9459-4c81-8646-15515c5439d9\") " pod="openshift-ingress-canary/ingress-canary-vxl2w" Nov 02 14:56:38.532167 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.532149 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l28\" (UniqueName: \"kubernetes.io/projected/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-api-access-q6l28\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.532491 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.532472 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbgl\" (UniqueName: \"kubernetes.io/projected/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kube-api-access-pxbgl\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.532678 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.532659 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7x7\" (UniqueName: \"kubernetes.io/projected/6fc4fca4-e664-4419-9ad8-a6930c959473-kube-api-access-lc7x7\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.532721 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.532700 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"periodic-gathering-tgkg9-9n6bd\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:56:38.537577 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.537561 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5xs\" (UniqueName: \"kubernetes.io/projected/2cdb8471-d93e-4750-9aef-984d1a738d4a-kube-api-access-6l5xs\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.538378 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.538360 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2bz\" (UniqueName: \"kubernetes.io/projected/46626844-9af4-499e-8f53-7428a1deb452-kube-api-access-hd2bz\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.538478 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.538460 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsqb\" (UniqueName: \"kubernetes.io/projected/683f8569-4473-4d34-ae00-ba1bf3bf9f5d-kube-api-access-5dsqb\") pod \"migrator-66df5d496b-l69rc\" (UID: \"683f8569-4473-4d34-ae00-ba1bf3bf9f5d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l69rc" Nov 02 14:56:38.538545 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.538529 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq7b\" (UniqueName: \"kubernetes.io/projected/91058aad-5292-4b62-ab2e-91444f80c847-kube-api-access-vkq7b\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.558708 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558690 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.558762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558718 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.558762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558735 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.558762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558754 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558777 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558813 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-hostroot\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558829 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-var-lib-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558830 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-ovn\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558830 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-host\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558855 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.558899 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.558880 4469 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558900 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-kubelet-config\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558884 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.558979 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:39.058964477 +0000 UTC m=+1.942566974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.558986 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-conf\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559009 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559033 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559064 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-slash\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559070 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-kubernetes\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559082 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559102 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559117 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559120 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559131 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559163 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559178 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-socket-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559190 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559198 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-netns\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559211 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysctl-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559213 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559227 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559238 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-kubelet\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559252 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559270 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559285 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559306 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-conf-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559322 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-sys\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559326 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559321 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559354 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559355 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559327 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cdb8471-d93e-4750-9aef-984d1a738d4a-hosts-file\") pod \"node-resolver-7x9dz\" (UID: \"2cdb8471-d93e-4750-9aef-984d1a738d4a\") " pod="openshift-dns/node-resolver-7x9dz" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559381 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-lib-modules\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559393 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559404 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-var-lib-kubelet\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559420 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559439 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559454 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559469 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559472 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-cnibin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559490 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-socket-dir-parent\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559491 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559515 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0bde169-ce7b-431d-a75e-fc6c56fbe779-crio-socket\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559520 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559519 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-netd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559529 4469 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559545 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559573 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559577 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-multus\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559532 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-cni-bin\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559591 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:39.059575172 +0000 UTC m=+1.943177684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : secret "metrics-daemon-secret" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559604 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-run-netns\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559628 4469 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559643 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559675 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:39.059662328 +0000 UTC m=+1.943264852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559690 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559692 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0959e584-488f-4332-a378-b927ac207e49-host\") pod \"node-ca-bzr9k\" (UID: \"0959e584-488f-4332-a378-b927ac207e49\") " pod="openshift-image-registry/node-ca-bzr9k" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559707 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559723 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559737 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559741 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-var-lib-cni-bin\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559744 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-etc-kubernetes\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559756 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559769 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-system-cni-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559775 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559790 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559815 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559821 4469 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:38.559872 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:39.059859256 +0000 UTC m=+1.943461753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559873 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09413fd8-4cfb-4e13-adb8-c8a58533ad5e-dbus\") pod \"global-pull-secret-syncer-tzjrs\" (UID: \"09413fd8-4cfb-4e13-adb8-c8a58533ad5e\") " pod="kube-system/global-pull-secret-syncer-tzjrs" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559812 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559887 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-modprobe-d\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559902 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559922 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.559896 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559933 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559941 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-run\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559961 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559981 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.559998 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560009 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-device-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560018 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560032 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-run-systemd\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560035 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560041 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-log-socket\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560055 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-host-kubelet\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560067 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-sysconfig\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560069 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-systemd-units\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560091 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560112 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560135 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560144 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91058aad-5292-4b62-ab2e-91444f80c847-etc-systemd\") pod \"tuned-gffg7\" (UID: \"91058aad-5292-4b62-ab2e-91444f80c847\") " pod="openshift-cluster-node-tuning-operator/tuned-gffg7" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560159 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560173 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-registration-dir\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560181 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-multus-certs\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560182 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560198 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6fc4fca4-e664-4419-9ad8-a6930c959473-host-slash\") pod \"iptables-alerter-b5jgk\" (UID: \"6fc4fca4-e664-4419-9ad8-a6930c959473\") " pod="openshift-network-operator/iptables-alerter-b5jgk" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560212 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-etc-openvswitch\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560217 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560239 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560257 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560278 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560282 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-system-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560295 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560300 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-sys-fs\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560307 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46626844-9af4-499e-8f53-7428a1deb452-node-log\") pod \"ovnkube-node-wpqbc\" (UID: \"46626844-9af4-499e-8f53-7428a1deb452\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560312 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560332 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560347 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02ba3151-5436-402c-87c6-47dbf8fbb3ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvz92\" (UID: \"02ba3151-5436-402c-87c6-47dbf8fbb3ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvz92" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560355 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-multus-cni-dir\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560358 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dfa937-57ef-4edc-b10d-2d995c93e764-cnibin\") pod \"multus-additional-cni-plugins-5lqlt\" (UID: \"51dfa937-57ef-4edc-b10d-2d995c93e764\") " pod="openshift-multus/multus-additional-cni-plugins-5lqlt" Nov 02 14:56:38.561305 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.560357 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8d1f65-2c06-4241-a3b4-025039c6e9e1-host-run-k8s-cni-cncf-io\") pod \"multus-65hmq\" (UID: \"5b8d1f65-2c06-4241-a3b4-025039c6e9e1\") " pod="openshift-multus/multus-65hmq" Nov 02 14:56:38.653882 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:38.653853 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzr9k_0959e584-488f-4332-a378-b927ac207e49/node-ca/0.log" Nov 02 14:56:39.063764 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.063694 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:39.063764 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.063731 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.063776 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063819 4469 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.063823 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063880 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.063863342 +0000 UTC m=+2.947465843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063897 4469 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063911 4469 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063929 4469 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063937 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs podName:50dc24f3-a26e-4583-bfe3-f1a25e97315a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.063926637 +0000 UTC m=+2.947529133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs") pod "network-metrics-daemon-245v8" (UID: "50dc24f3-a26e-4583-bfe3-f1a25e97315a") : secret "metrics-daemon-secret" not found Nov 02 14:56:39.063964 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063961 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.063947981 +0000 UTC m=+2.947550492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:39.064225 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:39.063986 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.063975412 +0000 UTC m=+2.947577923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:39.330330 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.330264 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:39.646645 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.646550 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vxl2w_a51a2287-9459-4c81-8646-15515c5439d9/serve-healthcheck-canary/0.log" Nov 02 14:56:39.812236 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:39.812207 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:40.020897 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.020850 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:40.047468 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.047442 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-l69rc_683f8569-4473-4d34-ae00-ba1bf3bf9f5d/migrator/0.log" Nov 02 14:56:40.071392 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.071362 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:40.071515 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.071405 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:40.071515 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.071429 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:40.071515 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.071446 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071515 4469 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071590 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:42.071574475 +0000 UTC m=+4.955176973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071518 4469 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071653 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:42.071641262 +0000 UTC m=+4.955243759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071522 4469 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:40.071688 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:40.071685 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:42.07167875 +0000 UTC m=+4.955281246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:40.072012 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.071855 4469 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:56:40.073940 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.073919 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dc24f3-a26e-4583-bfe3-f1a25e97315a-metrics-certs\") pod \"network-metrics-daemon-245v8\" (UID: \"50dc24f3-a26e-4583-bfe3-f1a25e97315a\") " pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:40.143702 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.143669 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245v8" Nov 02 14:56:40.224243 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.224178 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:40.315729 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.315662 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:40.563710 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.563679 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-245v8"] Nov 02 14:56:40.567790 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:40.567738 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dc24f3_a26e_4583_bfe3_f1a25e97315a.slice/crio-b5f4a482f65a8f508cb415941240cb7362cca0671e5b08578a7f59183c29e9ea WatchSource:0}: Error finding container b5f4a482f65a8f508cb415941240cb7362cca0671e5b08578a7f59183c29e9ea: Status 404 returned error can't find the container with id b5f4a482f65a8f508cb415941240cb7362cca0671e5b08578a7f59183c29e9ea Nov 02 14:56:40.570311 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:40.570293 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:56:41.081877 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.081847 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:41.275243 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.275204 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245v8" event={"ID":"50dc24f3-a26e-4583-bfe3-f1a25e97315a","Type":"ContainerStarted","Data":"b5f4a482f65a8f508cb415941240cb7362cca0671e5b08578a7f59183c29e9ea"} Nov 02 14:56:41.286209 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.286183 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:41.524934 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.524906 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:41.728345 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.728312 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:41.933659 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:41.933581 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:42.085272 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.085246 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:42.085385 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.085279 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:42.085385 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.085333 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:42.085385 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085359 4469 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:56:42.085597 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085415 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.085399387 +0000 UTC m=+8.969001900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : configmap "kube-rbac-proxy" not found Nov 02 14:56:42.085597 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085448 4469 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:42.085597 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085459 4469 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:42.085597 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085503 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.085491626 +0000 UTC m=+8.969094138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:42.085597 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:42.085520 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.085511208 +0000 UTC m=+8.969113710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:42.093039 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.093020 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:56:42.093236 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.093215 4469 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:56:42.132976 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.132928 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:42.279486 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.279453 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245v8" event={"ID":"50dc24f3-a26e-4583-bfe3-f1a25e97315a","Type":"ContainerStarted","Data":"954c371e47e717ed4a28d20212926cf5621989cc20b921720ca47fb5e4403ad3"} Nov 02 14:56:42.279627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:42.279526 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245v8" event={"ID":"50dc24f3-a26e-4583-bfe3-f1a25e97315a","Type":"ContainerStarted","Data":"bbd1ab839727c6191f16d75e973622343b69e75f7f9e78e3d9cefca9286898f0"} Nov 02 14:56:43.055181 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:43.055134 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:43.272606 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:43.272572 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:43.483364 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:43.483333 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:43.690549 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:43.690519 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:43.895469 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:43.895393 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:44.094935 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:44.094900 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:44.354469 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:44.354441 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:45.273764 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:45.273734 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:45.481435 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:45.481406 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:45.685024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:45.684951 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:45.888670 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:45.888635 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:46.092936 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:46.092901 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:46.110946 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:46.110921 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:46.111067 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:46.110953 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:46.111067 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:46.110995 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:46.111177 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:46.111079 4469 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:56:46.111177 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:46.111131 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls podName:f473c9a3-09d1-407c-be89-b00857fd2f56 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:54.111118101 +0000 UTC m=+16.994720603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls") pod "dns-default-w7g64" (UID: "f473c9a3-09d1-407c-be89-b00857fd2f56") : secret "dns-default-metrics-tls" not found Nov 02 14:56:46.111177 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:46.111081 4469 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:56:46.111334 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:56:46.111204 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls podName:f0bde169-ce7b-431d-a75e-fc6c56fbe779 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:54.111192985 +0000 UTC m=+16.994795481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4prgg" (UID: "f0bde169-ce7b-431d-a75e-fc6c56fbe779") : secret "insights-runtime-extractor-tls" not found Nov 02 14:56:46.111552 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:46.111530 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0bde169-ce7b-431d-a75e-fc6c56fbe779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:47.034261 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:47.034231 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.245058 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:47.245029 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.448325 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:47.448230 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.658124 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:47.658087 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.862944 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:47.862910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.069453 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:48.069422 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.275626 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:48.275595 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.512328 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:48.512299 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:56:48.527717 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:48.527654 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.729677 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:48.729643 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:49.269262 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:49.269234 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:49.498323 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:49.498287 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:49.704407 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:49.704331 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:49.912776 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:49.912739 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:50.117967 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:50.117934 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:50.465629 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:50.465601 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:51.913111 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:51.913081 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4crsd"] Nov 02 14:56:51.915229 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:51.915207 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:51.970463 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:51.970434 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 14:56:51.970623 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:51.970505 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 14:56:52.011454 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.011428 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gdg8q\"" Nov 02 14:56:52.020456 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.020433 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4crsd"] Nov 02 14:56:52.020743 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.020725 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dfdbf569-5d8rw"] Nov 02 14:56:52.022592 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.022578 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.047895 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.047877 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-tmp\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.048025 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.047903 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjk4s\" (UniqueName: \"kubernetes.io/projected/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-kube-api-access-hjk4s\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.076159 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.076059 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 14:56:52.076380 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.076278 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5rstk\"" Nov 02 14:56:52.076467 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.076446 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 14:56:52.076516 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.076494 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 14:56:52.077235 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.077213 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dfdbf569-5d8rw"] Nov 02 14:56:52.148287 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148257 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-tmp\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.148287 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148291 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjk4s\" (UniqueName: \"kubernetes.io/projected/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-kube-api-access-hjk4s\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.148522 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148315 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-image-registry-private-configuration\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148522 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148381 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjbr\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-kube-api-access-5zjbr\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148523 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9075214a-7dfa-44fd-9156-bb672e74a624-ca-trust-extracted\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148557 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-registry-certificates\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148627 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148576 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-trusted-ca\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148621 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-installation-pull-secrets\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148673 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-tmp\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.148762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148727 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-registry-tls\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.148762 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.148743 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-bound-sa-token\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.156117 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.156098 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 14:56:52.184854 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.184760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:52.209179 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.209157 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjk4s\" (UniqueName: \"kubernetes.io/projected/311c9cd3-8b65-4c41-82b7-f2d89f8dadc9-kube-api-access-hjk4s\") pod \"downloads-6dd789c8b7-4crsd\" (UID: \"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9\") " pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.227023 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.227001 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:56:52.249097 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249073 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-image-registry-private-configuration\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249183 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249103 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjbr\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-kube-api-access-5zjbr\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249183 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249122 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9075214a-7dfa-44fd-9156-bb672e74a624-ca-trust-extracted\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249183 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249143 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-registry-certificates\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249183 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249162 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-trusted-ca\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249183 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249177 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-installation-pull-secrets\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249401 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249202 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-registry-tls\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249401 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249224 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-bound-sa-token\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.249959 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.249931 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9075214a-7dfa-44fd-9156-bb672e74a624-ca-trust-extracted\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.256826 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.250630 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-registry-certificates\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.256826 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.250655 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9075214a-7dfa-44fd-9156-bb672e74a624-trusted-ca\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.256826 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.254673 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-image-registry-private-configuration\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.256826 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.254752 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9075214a-7dfa-44fd-9156-bb672e74a624-installation-pull-secrets\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.256826 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.255228 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-registry-tls\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.312898 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.312872 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-bound-sa-token\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.313026 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.312939 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjbr\" (UniqueName: \"kubernetes.io/projected/9075214a-7dfa-44fd-9156-bb672e74a624-kube-api-access-5zjbr\") pod \"image-registry-7dfdbf569-5d8rw\" (UID: \"9075214a-7dfa-44fd-9156-bb672e74a624\") " pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.332308 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.332288 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:52.363283 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.363260 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lw49d" Nov 02 14:56:52.388437 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.388411 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:52.606179 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.606149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:52.643349 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.643323 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4crsd"] Nov 02 14:56:52.646497 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:52.646476 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311c9cd3_8b65_4c41_82b7_f2d89f8dadc9.slice/crio-01df1237f61e05ce9710e23a77feb9b7e3c201dbbee38ab565bb2a8c2f950ad0 WatchSource:0}: Error finding container 01df1237f61e05ce9710e23a77feb9b7e3c201dbbee38ab565bb2a8c2f950ad0: Status 404 returned error can't find the container with id 01df1237f61e05ce9710e23a77feb9b7e3c201dbbee38ab565bb2a8c2f950ad0 Nov 02 14:56:52.701568 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.701543 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dfdbf569-5d8rw"] Nov 02 14:56:52.704926 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:52.704905 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9075214a_7dfa_44fd_9156_bb672e74a624.slice/crio-079737c27c74dce89c915a8a1bcb2348660cc79b4fadcf6ab68523b2331295fb WatchSource:0}: Error finding container 079737c27c74dce89c915a8a1bcb2348660cc79b4fadcf6ab68523b2331295fb: Status 404 returned error can't find the container with id 079737c27c74dce89c915a8a1bcb2348660cc79b4fadcf6ab68523b2331295fb Nov 02 14:56:52.832685 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:52.832649 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:53.309178 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.309140 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4crsd" event={"ID":"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9","Type":"ContainerStarted","Data":"01df1237f61e05ce9710e23a77feb9b7e3c201dbbee38ab565bb2a8c2f950ad0"} Nov 02 14:56:53.310552 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.310531 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" event={"ID":"9075214a-7dfa-44fd-9156-bb672e74a624","Type":"ContainerStarted","Data":"8c0106e5d436b603418be72bd5763a8c53e9c579093d10b5e25e9ec5cbd4d33b"} Nov 02 14:56:53.310633 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.310567 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" event={"ID":"9075214a-7dfa-44fd-9156-bb672e74a624","Type":"ContainerStarted","Data":"079737c27c74dce89c915a8a1bcb2348660cc79b4fadcf6ab68523b2331295fb"} Nov 02 14:56:53.310681 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.310659 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:56:53.430720 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.430674 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" podStartSLOduration=2.43066008 podStartE2EDuration="2.43066008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:53.430495429 +0000 UTC m=+16.314097947" watchObservedRunningTime="2026-11-02 14:56:53.43066008 +0000 UTC m=+16.314262596" Nov 02 14:56:53.442939 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.442910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:53.844025 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:53.843969 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:54.072264 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.072230 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:54.164507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.164413 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:54.164507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.164483 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:54.167214 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.167185 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0bde169-ce7b-431d-a75e-fc6c56fbe779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4prgg\" (UID: \"f0bde169-ce7b-431d-a75e-fc6c56fbe779\") " pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:54.167328 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.167290 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f473c9a3-09d1-407c-be89-b00857fd2f56-metrics-tls\") pod \"dns-default-w7g64\" (UID: \"f473c9a3-09d1-407c-be89-b00857fd2f56\") " pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:54.246448 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.246414 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:54.247507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.247480 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4prgg" Nov 02 14:56:54.283294 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.283110 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:54.629427 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.629342 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w7g64"] Nov 02 14:56:54.668787 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:54.668759 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf473c9a3_09d1_407c_be89_b00857fd2f56.slice/crio-b3ec1733e2f4b3b48715270dac2825ec7e049c274612cff5972f0870da03104a WatchSource:0}: Error finding container b3ec1733e2f4b3b48715270dac2825ec7e049c274612cff5972f0870da03104a: Status 404 returned error can't find the container with id b3ec1733e2f4b3b48715270dac2825ec7e049c274612cff5972f0870da03104a Nov 02 14:56:54.679462 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.679437 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4prgg"] Nov 02 14:56:54.682470 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:56:54.682446 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bde169_ce7b_431d_a75e_fc6c56fbe779.slice/crio-9799fc7470692217f110207d36bc5470c90e9294a7cc1c8257b5444b6fab9324 WatchSource:0}: Error finding container 9799fc7470692217f110207d36bc5470c90e9294a7cc1c8257b5444b6fab9324: Status 404 returned error can't find the container with id 9799fc7470692217f110207d36bc5470c90e9294a7cc1c8257b5444b6fab9324 Nov 02 14:56:54.969966 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:54.969935 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.172921 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:55.172882 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.323329 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:55.323079 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4prgg" event={"ID":"f0bde169-ce7b-431d-a75e-fc6c56fbe779","Type":"ContainerStarted","Data":"d3edac136d194c8cf57a7ac422d801dedf087be74b971fe1f507dec423c841c7"} Nov 02 14:56:55.323329 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:55.323207 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4prgg" event={"ID":"f0bde169-ce7b-431d-a75e-fc6c56fbe779","Type":"ContainerStarted","Data":"9799fc7470692217f110207d36bc5470c90e9294a7cc1c8257b5444b6fab9324"} Nov 02 14:56:55.325102 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:55.325065 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7g64" event={"ID":"f473c9a3-09d1-407c-be89-b00857fd2f56","Type":"ContainerStarted","Data":"b3ec1733e2f4b3b48715270dac2825ec7e049c274612cff5972f0870da03104a"} Nov 02 14:56:56.331842 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:56.331788 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4prgg" event={"ID":"f0bde169-ce7b-431d-a75e-fc6c56fbe779","Type":"ContainerStarted","Data":"dcdb091aaa706efa84680619b8362b503bd12af9d8c1b00da42a5ceeb18d3a64"} Nov 02 14:56:57.338676 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:57.337880 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7g64" event={"ID":"f473c9a3-09d1-407c-be89-b00857fd2f56","Type":"ContainerStarted","Data":"b3087aa8ed35031adeeab4589c32f55da69e023c705b3c5e39bfebc5253fbb94"} Nov 02 14:56:57.338676 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:57.337942 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w7g64" event={"ID":"f473c9a3-09d1-407c-be89-b00857fd2f56","Type":"ContainerStarted","Data":"85f9633a8f409a5b9fb4af7a3dc77202ece5d9597295b976fa485e256444818f"} Nov 02 14:56:57.338676 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:57.338581 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w7g64" Nov 02 14:56:58.243543 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:58.243504 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:58.345181 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:58.345148 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4prgg" event={"ID":"f0bde169-ce7b-431d-a75e-fc6c56fbe779","Type":"ContainerStarted","Data":"1fcc569830e501ccd345a5f182671510bed667b3c2f3382690a86c6eed10ff63"} Nov 02 14:56:58.448622 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:58.448576 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:59.240875 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:59.240841 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:59.444010 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:59.443974 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:59.651352 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:56:59.651258 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:02.654981 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:02.654933 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpqbc" Nov 02 14:57:03.069156 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:03.069124 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:03.274155 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:03.274123 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:04.714259 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:04.714217 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:05.473601 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:05.473564 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:05.692074 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:05.692023 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:05.897476 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:05.897402 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.025293 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.025255 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj"] Nov 02 14:57:06.029718 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.029693 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.082043 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.082021 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqtz2\"" Nov 02 14:57:06.112121 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.112085 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.131545 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.131517 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:57:06.131545 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.131526 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.131748 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.131560 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 14:57:06.131748 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.131663 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.131748 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.131732 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:57:06.132646 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.132623 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4"] Nov 02 14:57:06.135894 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.135873 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.152742 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.152690 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.152846 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.152774 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0071695-75e8-417e-a4c3-9ef622a53c5e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.152912 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.152863 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7mb\" (UniqueName: \"kubernetes.io/projected/c0071695-75e8-417e-a4c3-9ef622a53c5e-kube-api-access-vx7mb\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.152912 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.152897 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.182130 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.182078 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj"] Nov 02 14:57:06.182746 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.182723 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zflnb"] Nov 02 14:57:06.187135 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.186270 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.188511 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.188487 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4"] Nov 02 14:57:06.191235 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:57:06.191179 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:57:06.192442 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.192375 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:57:06.192736 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.192716 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 14:57:06.193890 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.192737 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zfbb7\"" Nov 02 14:57:06.232968 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.232936 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 14:57:06.239392 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.239358 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:57:06.253733 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253695 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.253870 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253751 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.253870 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253854 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kz7\" (UniqueName: \"kubernetes.io/projected/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-api-access-z4kz7\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253896 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0071695-75e8-417e-a4c3-9ef622a53c5e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253919 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253944 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.253969 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.254003 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7mb\" (UniqueName: \"kubernetes.io/projected/c0071695-75e8-417e-a4c3-9ef622a53c5e-kube-api-access-vx7mb\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.254022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.254023 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.254282 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.254049 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.255167 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.255140 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0071695-75e8-417e-a4c3-9ef622a53c5e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.256933 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.256909 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.257047 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.256960 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0071695-75e8-417e-a4c3-9ef622a53c5e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.291514 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.291481 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:57:06.315555 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.315524 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7mb\" (UniqueName: \"kubernetes.io/projected/c0071695-75e8-417e-a4c3-9ef622a53c5e-kube-api-access-vx7mb\") pod \"openshift-state-metrics-7bd7489957-6gxdj\" (UID: \"c0071695-75e8-417e-a4c3-9ef622a53c5e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.322829 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.322786 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:57:06.322963 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.322943 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zx8m7\"" Nov 02 14:57:06.334700 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.334672 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.347612 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.347580 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" Nov 02 14:57:06.355034 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355006 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kz7\" (UniqueName: \"kubernetes.io/projected/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-api-access-z4kz7\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355153 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355047 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355153 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355070 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-sys\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.355153 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355092 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355287 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355251 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355653 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355626 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355661 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-accelerators-collector-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355690 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355695 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355699 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355718 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-metrics-client-ca\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.355788 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355745 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-root\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.356131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355835 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.356131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355884 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-textfile\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.356131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355929 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-tls\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.356131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355951 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-wtmp\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.356131 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.355980 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bx4t\" (UniqueName: \"kubernetes.io/projected/611bc632-ddb5-4d92-a2af-9ee53d600c41-kube-api-access-8bx4t\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.356359 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.356272 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.358725 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.358560 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.358725 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.358648 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.419710 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.419642 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kz7\" (UniqueName: \"kubernetes.io/projected/eeb0b8b6-3e24-4b17-b733-2ff2d194a48f-kube-api-access-z4kz7\") pod \"kube-state-metrics-5dc8d7d89b-q4rx4\" (UID: \"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.452891 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.452859 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" Nov 02 14:57:06.456988 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.456957 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457113 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457004 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-accelerators-collector-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457113 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457045 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-metrics-client-ca\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457113 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457071 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-root\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457113 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457107 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-textfile\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457248 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457134 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-tls\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457248 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457158 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-wtmp\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457248 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457192 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bx4t\" (UniqueName: \"kubernetes.io/projected/611bc632-ddb5-4d92-a2af-9ee53d600c41-kube-api-access-8bx4t\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457248 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457238 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-sys\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457427 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457336 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-sys\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457599 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-wtmp\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457727 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-textfile\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.457899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457833 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/611bc632-ddb5-4d92-a2af-9ee53d600c41-root\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.458123 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.457901 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-accelerators-collector-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.458214 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.458189 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/611bc632-ddb5-4d92-a2af-9ee53d600c41-metrics-client-ca\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.459624 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.459588 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.460275 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.460248 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/611bc632-ddb5-4d92-a2af-9ee53d600c41-node-exporter-tls\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.520316 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.520250 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bx4t\" (UniqueName: \"kubernetes.io/projected/611bc632-ddb5-4d92-a2af-9ee53d600c41-kube-api-access-8bx4t\") pod \"node-exporter-zflnb\" (UID: \"611bc632-ddb5-4d92-a2af-9ee53d600c41\") " pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.563998 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.563968 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.776608 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.776561 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.776833 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.776791 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj"] Nov 02 14:57:06.779963 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:57:06.779921 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0071695_75e8_417e_a4c3_9ef622a53c5e.slice/crio-ffd4bd6ced70041690c50e160653bcba03b2036cba72944f18e3e42b136e7516 WatchSource:0}: Error finding container ffd4bd6ced70041690c50e160653bcba03b2036cba72944f18e3e42b136e7516: Status 404 returned error can't find the container with id ffd4bd6ced70041690c50e160653bcba03b2036cba72944f18e3e42b136e7516 Nov 02 14:57:06.803735 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.803711 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zflnb" Nov 02 14:57:06.819698 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:57:06.819670 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611bc632_ddb5_4d92_a2af_9ee53d600c41.slice/crio-35ecb3a983e77cca2cae895e3a7f96febf8b73a7547b51010e4021a8ccad593f WatchSource:0}: Error finding container 35ecb3a983e77cca2cae895e3a7f96febf8b73a7547b51010e4021a8ccad593f: Status 404 returned error can't find the container with id 35ecb3a983e77cca2cae895e3a7f96febf8b73a7547b51010e4021a8ccad593f Nov 02 14:57:06.833182 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.833160 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4"] Nov 02 14:57:06.836525 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:57:06.836497 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb0b8b6_3e24_4b17_b733_2ff2d194a48f.slice/crio-523fbaa446bfcdf9a5b9e779d99b45bbf3003792c58352b6713aa1953a4a3269 WatchSource:0}: Error finding container 523fbaa446bfcdf9a5b9e779d99b45bbf3003792c58352b6713aa1953a4a3269: Status 404 returned error can't find the container with id 523fbaa446bfcdf9a5b9e779d99b45bbf3003792c58352b6713aa1953a4a3269 Nov 02 14:57:06.986182 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:06.986149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.377860 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.377817 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zflnb" event={"ID":"611bc632-ddb5-4d92-a2af-9ee53d600c41","Type":"ContainerStarted","Data":"35ecb3a983e77cca2cae895e3a7f96febf8b73a7547b51010e4021a8ccad593f"} Nov 02 14:57:07.379179 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.379150 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" event={"ID":"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f","Type":"ContainerStarted","Data":"523fbaa446bfcdf9a5b9e779d99b45bbf3003792c58352b6713aa1953a4a3269"} Nov 02 14:57:07.381245 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.381215 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" event={"ID":"c0071695-75e8-417e-a4c3-9ef622a53c5e","Type":"ContainerStarted","Data":"4377da0f3d0c91e1332b066bdb74482f5e512d77918aefec7947d444fb642273"} Nov 02 14:57:07.381362 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.381273 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" event={"ID":"c0071695-75e8-417e-a4c3-9ef622a53c5e","Type":"ContainerStarted","Data":"0b736c7507fd157e4a3890ccf4fa92e09fbbb6bae743f059be1c84b8e1daee21"} Nov 02 14:57:07.381362 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.381305 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" event={"ID":"c0071695-75e8-417e-a4c3-9ef622a53c5e","Type":"ContainerStarted","Data":"ffd4bd6ced70041690c50e160653bcba03b2036cba72944f18e3e42b136e7516"} Nov 02 14:57:07.551597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.551565 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.756150 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.756105 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.969789 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:07.969758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:08.351433 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:08.350695 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w7g64" Nov 02 14:57:08.407238 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:08.407186 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:08.516163 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:08.516130 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wx98q" Nov 02 14:57:09.390251 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:09.390220 4469 generic.go:356] "Generic (PLEG): container finished" podID="611bc632-ddb5-4d92-a2af-9ee53d600c41" containerID="e1f15180eec45aee42416026ea7044fc2efea2f02b4c48c505cc6fe59c91900a" exitCode=0 Nov 02 14:57:09.390672 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:09.390257 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zflnb" event={"ID":"611bc632-ddb5-4d92-a2af-9ee53d600c41","Type":"ContainerDied","Data":"e1f15180eec45aee42416026ea7044fc2efea2f02b4c48c505cc6fe59c91900a"} Nov 02 14:57:10.303667 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.303627 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:10.396887 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.396847 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4crsd" event={"ID":"311c9cd3-8b65-4c41-82b7-f2d89f8dadc9","Type":"ContainerStarted","Data":"87e0c6c045e0be04b2af4ae07e849e1ea77b91487f5594b3f9b1c3f0b59f71f8"} Nov 02 14:57:10.397393 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.397308 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:57:10.400501 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.400469 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" event={"ID":"c0071695-75e8-417e-a4c3-9ef622a53c5e","Type":"ContainerStarted","Data":"be9c83f942c1e5fb13e4336f9b7a37de650b92b0d80160a76a625ade893dd282"} Nov 02 14:57:10.403045 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.403019 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zflnb" event={"ID":"611bc632-ddb5-4d92-a2af-9ee53d600c41","Type":"ContainerStarted","Data":"9e284490f0caf013aa4318a2e1d237edaabb96a0ba4154c2d3aba5b95a4dff46"} Nov 02 14:57:10.403176 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.403064 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zflnb" event={"ID":"611bc632-ddb5-4d92-a2af-9ee53d600c41","Type":"ContainerStarted","Data":"d585065b191bcd5bab3905d181f97ea2981f012276cfd4241824d3709ec46784"} Nov 02 14:57:10.405738 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.405625 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" event={"ID":"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f","Type":"ContainerStarted","Data":"44b67634d81a5f5dc57ad558ee0279faa205a566ccae6cc82593275877a20041"} Nov 02 14:57:10.405738 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.405674 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" event={"ID":"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f","Type":"ContainerStarted","Data":"ebd8c3a0b1412ea49b8d04f360fae34fa468a49cba2027e071cec9582b33e655"} Nov 02 14:57:10.405738 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.405702 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" event={"ID":"eeb0b8b6-3e24-4b17-b733-2ff2d194a48f","Type":"ContainerStarted","Data":"67b95c198f689dfbb3bbc396402759bc8d452292cef0ebce95a88dd9b2cc851a"} Nov 02 14:57:10.416867 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.416842 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4crsd" Nov 02 14:57:10.514460 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.514401 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4crsd" podStartSLOduration=2.373480427 podStartE2EDuration="19.514385103s" totalImagesPullingTime="17.140904676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:10.513831425 +0000 UTC m=+33.397433946" watchObservedRunningTime="2026-11-02 14:57:10.514385103 +0000 UTC m=+33.397987623" Nov 02 14:57:10.518824 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.518758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:10.634399 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.634291 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6gxdj" podStartSLOduration=3.206342753 podStartE2EDuration="5.634273375s" totalImagesPullingTime="2.427930622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:10.633304358 +0000 UTC m=+33.516906877" watchObservedRunningTime="2026-11-02 14:57:10.634273375 +0000 UTC m=+33.517875893" Nov 02 14:57:10.719783 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.719746 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:10.754084 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.753585 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q4rx4" podStartSLOduration=2.196536333 podStartE2EDuration="4.753565721s" totalImagesPullingTime="2.557029388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:10.75261969 +0000 UTC m=+33.636222211" watchObservedRunningTime="2026-11-02 14:57:10.753565721 +0000 UTC m=+33.637168241" Nov 02 14:57:10.872319 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.872256 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zflnb" podStartSLOduration=2.661463662 podStartE2EDuration="4.872240863s" totalImagesPullingTime="2.122551123s" totalInitContainerRuntime="88.226078ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:10.871154842 +0000 UTC m=+33.754757362" watchObservedRunningTime="2026-11-02 14:57:10.872240863 +0000 UTC m=+33.755843382" Nov 02 14:57:10.926825 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:10.926722 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:11.148978 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.148941 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:11.837037 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.836999 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85b96b6b66-ck9sh"] Nov 02 14:57:11.869554 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.869528 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.884918 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.884891 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85b96b6b66-ck9sh"] Nov 02 14:57:11.906020 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.905992 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-metrics-client-ca\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906133 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906026 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906133 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906051 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvvm\" (UniqueName: \"kubernetes.io/projected/ead863a6-a542-46bb-828e-6fe9ebba5fa5-kube-api-access-nnvvm\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906133 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906113 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906285 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906183 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906285 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906249 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-federate-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906374 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906307 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.906439 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.906413 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-serving-certs-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:11.933389 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933359 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 14:57:11.933542 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933448 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 14:57:11.933542 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933471 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 14:57:11.933671 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933547 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 14:57:11.933671 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933651 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jkvkg\"" Nov 02 14:57:11.933778 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:11.933765 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 14:57:12.006986 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.006951 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-serving-certs-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.006995 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-metrics-client-ca\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007017 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007044 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvvm\" (UniqueName: \"kubernetes.io/projected/ead863a6-a542-46bb-828e-6fe9ebba5fa5-kube-api-access-nnvvm\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007077 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007104 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007138 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007141 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-federate-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.007476 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.007187 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.008103 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.008017 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-metrics-client-ca\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.008446 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.008335 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-serving-certs-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.010340 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.010313 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.010510 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.010488 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.011526 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.011504 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-federate-client-tls\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.011526 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.011520 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead863a6-a542-46bb-828e-6fe9ebba5fa5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.021622 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.021580 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 14:57:12.028830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.028786 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead863a6-a542-46bb-828e-6fe9ebba5fa5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.069004 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.068977 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvvm\" (UniqueName: \"kubernetes.io/projected/ead863a6-a542-46bb-828e-6fe9ebba5fa5-kube-api-access-nnvvm\") pod \"telemeter-client-85b96b6b66-ck9sh\" (UID: \"ead863a6-a542-46bb-828e-6fe9ebba5fa5\") " pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.236998 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.236964 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" Nov 02 14:57:12.339676 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.339595 4469 patch_prober.go:29] interesting pod/image-registry-7dfdbf569-5d8rw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 14:57:12.339818 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.339666 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" podUID="9075214a-7dfa-44fd-9156-bb672e74a624" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:12.757581 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:12.757548 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85b96b6b66-ck9sh"] Nov 02 14:57:12.761825 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:57:12.761779 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead863a6_a542_46bb_828e_6fe9ebba5fa5.slice/crio-c4ca5cc764862bbdd4db15a053fa913b4aae76798a8d4fa6007a69bd7f42cbcf WatchSource:0}: Error finding container c4ca5cc764862bbdd4db15a053fa913b4aae76798a8d4fa6007a69bd7f42cbcf: Status 404 returned error can't find the container with id c4ca5cc764862bbdd4db15a053fa913b4aae76798a8d4fa6007a69bd7f42cbcf Nov 02 14:57:13.065341 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:13.065271 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:13.264465 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:13.264422 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:13.425184 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:13.425098 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" event={"ID":"ead863a6-a542-46bb-828e-6fe9ebba5fa5","Type":"ContainerStarted","Data":"c4ca5cc764862bbdd4db15a053fa913b4aae76798a8d4fa6007a69bd7f42cbcf"} Nov 02 14:57:14.322989 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:14.322957 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dfdbf569-5d8rw" Nov 02 14:57:14.709190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:14.709125 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:14.914429 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:14.914396 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.146502 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.146417 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.274113 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.274079 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:57:15.293060 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.293036 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.324204 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.324177 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.336596 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.336790 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.336880 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.336971 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337007 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337094 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337139 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337195 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337356 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337386 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwz29\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337584 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337615 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337851 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.337899 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338117 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338170 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338419 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338455 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338493 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.341827 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.338711 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.358106 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.358073 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.439494 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439413 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439494 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439463 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439494 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439492 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439517 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439545 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439574 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439609 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439635 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439668 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439690 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwz29\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439715 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439738 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.439780 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439767 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439821 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439853 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439893 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439936 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439961 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.439992 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.440026 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440310 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.440240 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.440847 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.440829 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.442222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.442188 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.444668 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444644 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 14:57:15.444768 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444707 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444790 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444871 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444887 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444962 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444970 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.444970 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 14:57:15.445078 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.445032 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 14:57:15.445439 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.445193 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bmlmk\"" Nov 02 14:57:15.445439 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.445350 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 14:57:15.450509 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.450486 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.450673 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.450652 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.451124 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.451096 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.451784 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.451759 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.452951 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.452899 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.453453 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.453425 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.454098 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.453949 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.454098 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.454064 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.454833 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.454791 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.455177 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.455154 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.472034 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472009 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 14:57:15.472127 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472028 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 14:57:15.472127 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472048 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 14:57:15.472127 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472067 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 14:57:15.472270 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472132 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8h8jnfvrd3jvo\"" Nov 02 14:57:15.472270 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.472181 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 14:57:15.481010 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.480967 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.482906 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.482863 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.483036 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.482944 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.483465 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.483103 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.483465 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.483425 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.483832 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.483791 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.500961 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.500935 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwz29\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\") pod \"prometheus-k8s-0\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:15.744808 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:15.744762 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:16.368674 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:16.368605 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:57:16.372150 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:57:16.372110 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb965aa17_035c_4c6e_923f_bed9ae555a6e.slice/crio-9b6dfa261c017453693bb99b8679cd77618c0c363244cbb356bda0b330657303 WatchSource:0}: Error finding container 9b6dfa261c017453693bb99b8679cd77618c0c363244cbb356bda0b330657303: Status 404 returned error can't find the container with id 9b6dfa261c017453693bb99b8679cd77618c0c363244cbb356bda0b330657303 Nov 02 14:57:16.439374 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:16.439336 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" event={"ID":"ead863a6-a542-46bb-828e-6fe9ebba5fa5","Type":"ContainerStarted","Data":"a23450fe91366180f26c2b12695cadab2dafb50ae3e009f81882f78d466662de"} Nov 02 14:57:16.440818 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:16.440775 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"9b6dfa261c017453693bb99b8679cd77618c0c363244cbb356bda0b330657303"} Nov 02 14:57:17.192597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:17.192561 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.406040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:17.401886 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.616369 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:17.616304 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.451571 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:18.451532 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690" exitCode=0 Nov 02 14:57:18.451989 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:18.451570 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690"} Nov 02 14:57:18.454903 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:18.454871 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" event={"ID":"ead863a6-a542-46bb-828e-6fe9ebba5fa5","Type":"ContainerStarted","Data":"26fed7eea5d47ef4319116a84f93247c6444dfdf6737d1760618072d06ee0161"} Nov 02 14:57:18.455006 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:18.454924 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" event={"ID":"ead863a6-a542-46bb-828e-6fe9ebba5fa5","Type":"ContainerStarted","Data":"4aa92621348b70a7c1233e46aa4626d7121482da11e441131393213be8c7400f"} Nov 02 14:57:19.582534 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:19.582331 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85b96b6b66-ck9sh" podStartSLOduration=3.193992513 podStartE2EDuration="8.582317174s" totalImagesPullingTime="5.388324661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:19.580841346 +0000 UTC m=+42.464443857" watchObservedRunningTime="2026-11-02 14:57:19.582317174 +0000 UTC m=+42.465919692" Nov 02 14:57:21.259210 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:21.259158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:21.467040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:21.467009 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.474332 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:22.474301 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678"} Nov 02 14:57:22.882661 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:22.882581 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.111839 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:23.111808 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.479401 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:23.479371 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4"} Nov 02 14:57:25.489308 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:25.489279 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9"} Nov 02 14:57:25.489585 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:25.489344 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3"} Nov 02 14:57:26.495444 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:26.495403 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5"} Nov 02 14:57:26.495903 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:26.495465 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerStarted","Data":"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33"} Nov 02 14:57:26.785185 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:26.785060 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.45210393 podStartE2EDuration="11.785040276s" totalImagesPullingTime="8.303987717s" totalInitContainerRuntime="28.948629ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:26.747512705 +0000 UTC m=+49.631115223" watchObservedRunningTime="2026-11-02 14:57:26.785040276 +0000 UTC m=+49.668642796" Nov 02 14:57:26.864244 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:26.864209 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.078085 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:27.078009 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.458892 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:27.458821 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.665066 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:27.665035 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.973931 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:28.973900 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:29.186323 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:29.186286 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.745384 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:30.745347 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:35.397398 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:35.397335 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:35.601380 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:35.601346 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.186516 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:57:36.186479 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:57:39.847899 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:39.847860 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:57:39.848298 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:39.848115 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" podUID="8e14b273-2cc6-4872-88e3-fec55e2b0631" containerName="insights-gathering" containerID="cri-o://45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97" gracePeriod=30 Nov 02 14:57:40.142203 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.142174 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:57:40.263723 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.263691 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") pod \"8e14b273-2cc6-4872-88e3-fec55e2b0631\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " Nov 02 14:57:40.263909 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.263790 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") pod \"8e14b273-2cc6-4872-88e3-fec55e2b0631\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " Nov 02 14:57:40.263909 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.263840 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") pod \"8e14b273-2cc6-4872-88e3-fec55e2b0631\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " Nov 02 14:57:40.263909 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.263872 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") pod \"8e14b273-2cc6-4872-88e3-fec55e2b0631\" (UID: \"8e14b273-2cc6-4872-88e3-fec55e2b0631\") " Nov 02 14:57:40.264193 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.264171 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle" pod "8e14b273-2cc6-4872-88e3-fec55e2b0631" (UID: "8e14b273-2cc6-4872-88e3-fec55e2b0631"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:40.264241 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.264186 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle" pod "8e14b273-2cc6-4872-88e3-fec55e2b0631" (UID: "8e14b273-2cc6-4872-88e3-fec55e2b0631"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:40.264284 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.264265 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path" pod "8e14b273-2cc6-4872-88e3-fec55e2b0631" (UID: "8e14b273-2cc6-4872-88e3-fec55e2b0631"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:40.266225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.266201 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9" pod "8e14b273-2cc6-4872-88e3-fec55e2b0631" (UID: "8e14b273-2cc6-4872-88e3-fec55e2b0631"). InnerVolumeSpecName "kube-api-access-glcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:40.364572 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.364499 4469 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-service-ca-bundle\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:57:40.364572 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.364531 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcf9\" (UniqueName: \"kubernetes.io/projected/8e14b273-2cc6-4872-88e3-fec55e2b0631-kube-api-access-glcf9\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:57:40.364572 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.364543 4469 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8e14b273-2cc6-4872-88e3-fec55e2b0631-archives-path\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:57:40.364572 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.364552 4469 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e14b273-2cc6-4872-88e3-fec55e2b0631-trusted-ca-bundle\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:57:40.544266 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.544229 4469 generic.go:356] "Generic (PLEG): container finished" podID="8e14b273-2cc6-4872-88e3-fec55e2b0631" containerID="45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97" exitCode=2 Nov 02 14:57:40.544440 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.544269 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" Nov 02 14:57:40.544440 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.544285 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" event={"ID":"8e14b273-2cc6-4872-88e3-fec55e2b0631","Type":"ContainerDied","Data":"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97"} Nov 02 14:57:40.544440 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.544335 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tgkg9-9n6bd" event={"ID":"8e14b273-2cc6-4872-88e3-fec55e2b0631","Type":"ContainerDied","Data":"d5ec43199f1657b0b69f84f54c701fb7859ce2adec5836475e822dddb2c10614"} Nov 02 14:57:40.544440 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.544357 4469 scope.go:122] "RemoveContainer" containerID="45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97" Nov 02 14:57:40.556071 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.556050 4469 scope.go:122] "RemoveContainer" containerID="45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97" Nov 02 14:57:40.556325 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:57:40.556308 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97\": container with ID starting with 45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97 not found: ID does not exist" containerID="45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97" Nov 02 14:57:40.556373 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.556333 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97"} err="failed to get container status \"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97\": rpc error: code = NotFound desc = could not find container \"45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97\": container with ID starting with 45aef4e0df12a48bbe48a8f03a50b2904a92679baa523c48ad528e5325479d97 not found: ID does not exist" Nov 02 14:57:40.707120 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.707046 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:57:40.709555 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.709535 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-tgkg9-9n6bd"] Nov 02 14:57:40.867908 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:40.867873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.074225 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:41.074191 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.235475 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:41.235443 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e14b273-2cc6-4872-88e3-fec55e2b0631" path="/var/lib/kubelet/pods/8e14b273-2cc6-4872-88e3-fec55e2b0631/volumes" Nov 02 14:57:43.196206 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:43.196165 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:46.164503 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:46.164463 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:46.217720 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:46.217694 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:46.580636 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:46.580607 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:57:54.949119 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:54.949084 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.155548 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:55.155509 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.752030 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:55.751996 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.958332 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:55.958296 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.164433 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:57:56.164358 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:05.326129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:05.326096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:05.525474 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:05.525431 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:06.184147 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:06.184111 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:58:08.271022 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.270984 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:08.271473 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271280 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="prometheus" containerID="cri-o://d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" gracePeriod=600 Nov 02 14:58:08.271473 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271445 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-web" containerID="cri-o://312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9" gracePeriod=600 Nov 02 14:58:08.271592 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271480 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy" containerID="cri-o://6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33" gracePeriod=600 Nov 02 14:58:08.271592 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271521 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="thanos-sidecar" containerID="cri-o://7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3" gracePeriod=600 Nov 02 14:58:08.271592 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271576 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="config-reloader" containerID="cri-o://5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4" gracePeriod=600 Nov 02 14:58:08.271746 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.271643 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5" gracePeriod=600 Nov 02 14:58:08.656772 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656693 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33" exitCode=0 Nov 02 14:58:08.656772 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656716 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3" exitCode=0 Nov 02 14:58:08.656772 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656724 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4" exitCode=0 Nov 02 14:58:08.657040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656773 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33"} Nov 02 14:58:08.657040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656865 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3"} Nov 02 14:58:08.657040 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:08.656912 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4"} Nov 02 14:58:09.663769 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:09.663673 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5" exitCode=0 Nov 02 14:58:09.663769 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:09.663707 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9" exitCode=0 Nov 02 14:58:09.663769 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:09.663745 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5"} Nov 02 14:58:09.664195 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:09.663856 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9"} Nov 02 14:58:10.869382 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:10.869344 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:11.067940 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:11.067906 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:11.566648 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:11.566602 4469 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:11.566865 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:11.566661 4469 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:11.566865 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:11.566680 4469 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:11.566865 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:11.566695 4469 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Nov 02 14:58:14.135146 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.135120 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.248525 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248502 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248537 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248568 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248595 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248631 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248658 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.248694 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248693 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248721 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248760 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248795 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248852 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248887 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248917 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248955 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.248998 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249035 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249029 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249058 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249090 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249092 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249134 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249166 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\") pod \"b965aa17-035c-4c6e-923f-bed9ae555a6e\" (UID: \"b965aa17-035c-4c6e-923f-bed9ae555a6e\") " Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249372 4469 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.249470 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249393 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.249850 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.249767 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.250244 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.250143 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.250366 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.250311 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.250546 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.250498 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.250830 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.250788 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:14.252178 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.251619 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.252178 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.252136 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:14.252548 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.252493 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.252548 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.252511 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:14.252688 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.252556 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.253059 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.253031 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.253216 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.253197 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.253507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.253474 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.253609 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.253588 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "kube-api-access-hwz29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:14.254207 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.254186 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.254492 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.254474 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.254837 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.254818 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:14.263546 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.263507 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config" pod "b965aa17-035c-4c6e-923f-bed9ae555a6e" (UID: "b965aa17-035c-4c6e-923f-bed9ae555a6e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:14.350155 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350129 4469 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-grpc-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350159 4469 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-db\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350175 4469 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-tls-assets\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350191 4469 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350207 4469 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350222 4469 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-web-config\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350235 4469 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b965aa17-035c-4c6e-923f-bed9ae555a6e-config-out\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350250 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwz29\" (UniqueName: \"kubernetes.io/projected/b965aa17-035c-4c6e-923f-bed9ae555a6e-kube-api-access-hwz29\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350269 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350264 4469 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350281 4469 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350296 4469 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350313 4469 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350328 4469 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-configmap-metrics-client-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350342 4469 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-metrics-client-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350356 4469 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b965aa17-035c-4c6e-923f-bed9ae555a6e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350364 4469 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350374 4469 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-kube-rbac-proxy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350382 4469 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-config\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.350524 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.350402 4469 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b965aa17-035c-4c6e-923f-bed9ae555a6e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 14:58:14.686129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.686048 4469 generic.go:356] "Generic (PLEG): container finished" podID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" exitCode=0 Nov 02 14:58:14.686129 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.686112 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678"} Nov 02 14:58:14.686300 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.686131 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.686300 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.686191 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b965aa17-035c-4c6e-923f-bed9ae555a6e","Type":"ContainerDied","Data":"9b6dfa261c017453693bb99b8679cd77618c0c363244cbb356bda0b330657303"} Nov 02 14:58:14.686300 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.686259 4469 scope.go:122] "RemoveContainer" containerID="76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5" Nov 02 14:58:14.694730 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.694656 4469 scope.go:122] "RemoveContainer" containerID="6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33" Nov 02 14:58:14.701147 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.701127 4469 scope.go:122] "RemoveContainer" containerID="312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9" Nov 02 14:58:14.707281 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.707263 4469 scope.go:122] "RemoveContainer" containerID="7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3" Nov 02 14:58:14.713437 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.713420 4469 scope.go:122] "RemoveContainer" containerID="5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4" Nov 02 14:58:14.719600 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.719582 4469 scope.go:122] "RemoveContainer" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" Nov 02 14:58:14.725954 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.725938 4469 scope.go:122] "RemoveContainer" containerID="7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690" Nov 02 14:58:14.731936 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.731866 4469 scope.go:122] "RemoveContainer" containerID="76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5" Nov 02 14:58:14.732129 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.732102 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5\": container with ID starting with 76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5 not found: ID does not exist" containerID="76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5" Nov 02 14:58:14.732218 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732136 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5"} err="failed to get container status \"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5\": rpc error: code = NotFound desc = could not find container \"76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5\": container with ID starting with 76d43be0fe20781fee1aa193a17e3c9412dc9eb6d4507ac48ca8a1aedd4db2d5 not found: ID does not exist" Nov 02 14:58:14.732218 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732162 4469 scope.go:122] "RemoveContainer" containerID="6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33" Nov 02 14:58:14.732433 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.732414 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33\": container with ID starting with 6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33 not found: ID does not exist" containerID="6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33" Nov 02 14:58:14.732477 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732440 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33"} err="failed to get container status \"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33\": rpc error: code = NotFound desc = could not find container \"6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33\": container with ID starting with 6c4657d1952770158b535efb310656adfd473ada98888b30929aa10a1c42ee33 not found: ID does not exist" Nov 02 14:58:14.732477 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732459 4469 scope.go:122] "RemoveContainer" containerID="312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9" Nov 02 14:58:14.732712 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.732694 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9\": container with ID starting with 312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9 not found: ID does not exist" containerID="312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9" Nov 02 14:58:14.732753 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732719 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9"} err="failed to get container status \"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9\": rpc error: code = NotFound desc = could not find container \"312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9\": container with ID starting with 312869746cce1ac6e299ecbb2fdd844db3e35783fa0700546901c524f3e4ecc9 not found: ID does not exist" Nov 02 14:58:14.732753 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.732735 4469 scope.go:122] "RemoveContainer" containerID="7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3" Nov 02 14:58:14.733021 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.733001 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3\": container with ID starting with 7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3 not found: ID does not exist" containerID="7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3" Nov 02 14:58:14.733066 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733026 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3"} err="failed to get container status \"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3\": rpc error: code = NotFound desc = could not find container \"7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3\": container with ID starting with 7256c4e03126cbc398342e547267da4e446e90c8845924c492122ec5db0760b3 not found: ID does not exist" Nov 02 14:58:14.733066 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733043 4469 scope.go:122] "RemoveContainer" containerID="5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4" Nov 02 14:58:14.733240 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.733224 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4\": container with ID starting with 5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4 not found: ID does not exist" containerID="5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4" Nov 02 14:58:14.733278 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733244 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4"} err="failed to get container status \"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4\": rpc error: code = NotFound desc = could not find container \"5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4\": container with ID starting with 5274ae5191bd1e96a7acc4cd59f38aaa05675bfdc7ef68a5a8673032bb0028f4 not found: ID does not exist" Nov 02 14:58:14.733278 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733256 4469 scope.go:122] "RemoveContainer" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" Nov 02 14:58:14.733435 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.733420 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678\": container with ID starting with d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678 not found: ID does not exist" containerID="d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678" Nov 02 14:58:14.733467 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733437 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678"} err="failed to get container status \"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678\": rpc error: code = NotFound desc = could not find container \"d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678\": container with ID starting with d20c42ed3ee21a7e055a0549d9702aa6e7b4c91032199b2db39909050dc95678 not found: ID does not exist" Nov 02 14:58:14.733467 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733448 4469 scope.go:122] "RemoveContainer" containerID="7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690" Nov 02 14:58:14.733635 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:14.733617 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690\": container with ID starting with 7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690 not found: ID does not exist" containerID="7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690" Nov 02 14:58:14.733676 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.733638 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690"} err="failed to get container status \"7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690\": rpc error: code = NotFound desc = could not find container \"7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690\": container with ID starting with 7a37a3abb6cbaa7444aee3853e2052edc0e11be8c6c26f9410d0a700549e2690 not found: ID does not exist" Nov 02 14:58:14.854359 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.854325 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:14.905096 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.905035 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:14.905728 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.905707 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:14.905981 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.905969 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.905983 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.905997 4469 cpu_manager.go:414] "containerMap: removing container" podUID="8e14b273-2cc6-4872-88e3-fec55e2b0631" containerName="insights-gathering" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906002 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e14b273-2cc6-4872-88e3-fec55e2b0631" containerName="insights-gathering" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906010 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="init-config-reloader" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906015 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="init-config-reloader" Nov 02 14:58:14.906024 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906023 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="prometheus" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906028 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="prometheus" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906034 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="thanos-sidecar" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906038 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="thanos-sidecar" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906048 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-web" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906053 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-web" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906060 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906064 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906070 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="config-reloader" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906075 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="config-reloader" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906120 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906128 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="config-reloader" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906138 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906143 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e14b273-2cc6-4872-88e3-fec55e2b0631" containerName="insights-gathering" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906149 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="thanos-sidecar" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906154 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="prometheus" Nov 02 14:58:14.906190 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.906160 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" containerName="kube-rbac-proxy-web" Nov 02 14:58:14.909576 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.909559 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.955988 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.955894 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.955988 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.955959 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956068 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956098 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956123 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956222 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956151 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956365 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956226 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956365 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956273 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956365 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956313 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956365 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956344 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956365 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956362 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956378 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956413 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956470 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956496 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956527 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956559 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956597 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956579 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrtg\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-kube-api-access-4hrtg\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956865 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956626 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.956865 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.956690 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:14.962970 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.962940 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 14:58:14.963196 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963174 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:14.963389 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963208 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 14:58:14.963389 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963215 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 14:58:14.963389 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963208 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963439 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963448 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963459 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963475 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963477 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963440 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bmlmk\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963496 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963498 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963524 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8h8jnfvrd3jvo\"" Nov 02 14:58:14.963632 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:14.963555 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 14:58:15.013593 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.013553 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 14:58:15.013720 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.013627 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 14:58:15.031659 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.031101 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:15.058142 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058100 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058142 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058142 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058349 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058159 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058349 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058182 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058349 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058306 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058353 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058435 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058511 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058562 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058591 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058615 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058673 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.058709 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.058694 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrtg\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-kube-api-access-4hrtg\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059344 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059213 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059344 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059256 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059344 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059321 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059344 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059328 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059582 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059361 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.059582 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059449 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.061385 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.059714 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.062075 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.061853 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.062075 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.061947 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.062075 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.061984 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.062075 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.062015 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063197 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.062583 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063197 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.062669 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063197 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.062917 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063197 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.062968 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063443 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.063393 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063499 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.063462 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.063843 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.063819 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01f9099a-203c-4431-96e0-8c34baf619ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.064041 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.063989 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.064380 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.064344 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01f9099a-203c-4431-96e0-8c34baf619ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.064985 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.064952 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.065087 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.065064 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.065270 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.065245 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.065517 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.065496 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-config\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.065898 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.065878 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.065998 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.065983 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01f9099a-203c-4431-96e0-8c34baf619ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.122706 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.122669 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrtg\" (UniqueName: \"kubernetes.io/projected/01f9099a-203c-4431-96e0-8c34baf619ae-kube-api-access-4hrtg\") pod \"prometheus-k8s-0\" (UID: \"01f9099a-203c-4431-96e0-8c34baf619ae\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.226174 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.226138 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:15.236386 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.236363 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b965aa17-035c-4c6e-923f-bed9ae555a6e" path="/var/lib/kubelet/pods/b965aa17-035c-4c6e-923f-bed9ae555a6e/volumes" Nov 02 14:58:15.659261 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.659188 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:15.662529 ip-10-0-135-103 kubenswrapper[4469]: W1102 14:58:15.662500 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f9099a_203c_4431_96e0_8c34baf619ae.slice/crio-ac0ec822a8b86d6f473f10421b661bb1801d2cc38a2ab898c4f280e43058012f WatchSource:0}: Error finding container ac0ec822a8b86d6f473f10421b661bb1801d2cc38a2ab898c4f280e43058012f: Status 404 returned error can't find the container with id ac0ec822a8b86d6f473f10421b661bb1801d2cc38a2ab898c4f280e43058012f Nov 02 14:58:15.692025 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:15.692000 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"ac0ec822a8b86d6f473f10421b661bb1801d2cc38a2ab898c4f280e43058012f"} Nov 02 14:58:16.195102 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:16.195065 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:16.700690 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:16.700647 4469 generic.go:356] "Generic (PLEG): container finished" podID="01f9099a-203c-4431-96e0-8c34baf619ae" containerID="7e69653c4d065876b20dd6de28f2f95a5b7c01416344699dd4e61bfbcf356daf" exitCode=0 Nov 02 14:58:16.701100 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:16.700695 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerDied","Data":"7e69653c4d065876b20dd6de28f2f95a5b7c01416344699dd4e61bfbcf356daf"} Nov 02 14:58:17.708068 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708030 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"02023a52a7a5e12e5814c3458ddb59ac031b7df29e71eae160f59be0d4b8fa42"} Nov 02 14:58:17.708472 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708097 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"0d6aef1ae9a3a9f456cb722257054a83b17c5928757a9e76e95bcf7c2f6a3e01"} Nov 02 14:58:17.708472 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708125 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"22a84dd4e3bb02ff67339c511756c1357a497c2a6b259d7825c43dbed4bc2322"} Nov 02 14:58:17.708472 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708148 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"6d930880a7958b4d3095b0cdb8810519926629c0b8fd52e7bebc13a9e6991b6b"} Nov 02 14:58:17.708472 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708177 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"4585e64e7da97d4419e306e848e6141a84aa6a2acbf1cb1080f01bd855beea39"} Nov 02 14:58:17.708472 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.708199 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01f9099a-203c-4431-96e0-8c34baf619ae","Type":"ContainerStarted","Data":"b1d471bbba11e43ce92a96f510db98da2dcdefad427ef228ae6e8866f9036a85"} Nov 02 14:58:17.841658 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:17.841587 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.825582932 podStartE2EDuration="3.841562453s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.979521ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:17.840628056 +0000 UTC m=+100.724230580" watchObservedRunningTime="2026-11-02 14:58:17.841562453 +0000 UTC m=+100.725164974" Nov 02 14:58:20.227166 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:20.227118 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:23.239154 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:23.239118 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:23.441495 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:23.441459 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:36.185130 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:58:36.185090 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:58:42.307378 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:58:42.307341 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:04.004175 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:04.004140 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:06.184459 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:59:06.184426 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:59:10.594461 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:10.594425 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:10.818416 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:10.818385 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:15.227233 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:15.227193 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:15.242507 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:15.242476 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:15.932934 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:15.932908 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:17.359438 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:17.359406 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:17.565289 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:17.565251 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:23.583435 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:23.583395 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:27.494960 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:27.494921 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:28.489880 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:28.489839 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:36.184010 ip-10-0-135-103 kubenswrapper[4469]: E1102 14:59:36.183969 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 14:59:40.399086 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:40.399049 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:40.607886 ip-10-0-135-103 kubenswrapper[4469]: I1102 14:59:40.607851 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:06.184112 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:00:06.184064 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:00:06.653119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:06.653079 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:06.870833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:06.870784 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:08.354100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:08.354067 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:08.571360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:08.571319 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:09.703389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:09.703355 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:14.473553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:14.473517 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:14.680460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:14.680427 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:15.989044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:15.989012 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:16.217830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:16.217782 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:16.447136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:16.447057 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:19.956627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:19.956599 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:23.959159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:23.959126 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:36.184583 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:00:36.184539 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:00:47.061454 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:47.061407 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:47.318640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:47.318558 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:52.169544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:52.169504 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:53.396617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:53.396584 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:59.566374 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:00:59.566328 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:01.816853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:01.816818 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:02.054376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:02.054340 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:03.229981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.229935 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:03.490346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.490271 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v"] Nov 02 15:01:03.492539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.492522 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.505652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.505631 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:03.570782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.570758 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 15:01:03.570897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.570784 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 15:01:03.582745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582722 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.582842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582751 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.582842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582789 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.582928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582844 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.582963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582938 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.583005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.582964 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8pf\" (UniqueName: \"kubernetes.io/projected/5eb74ec9-129c-44bf-8b1d-221f096eb55f-kube-api-access-st8pf\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.584171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.584157 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 15:01:03.584457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.584442 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5tklz\"" Nov 02 15:01:03.584529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.584514 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 15:01:03.584579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.584534 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 15:01:03.609530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.609478 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v"] Nov 02 15:01:03.684257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684233 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8pf\" (UniqueName: \"kubernetes.io/projected/5eb74ec9-129c-44bf-8b1d-221f096eb55f-kube-api-access-st8pf\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684275 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684297 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684335 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684350 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684373 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.684736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.684719 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.686649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.686623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb74ec9-129c-44bf-8b1d-221f096eb55f-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.686754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.686732 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.686915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.686898 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.686915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.686905 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5eb74ec9-129c-44bf-8b1d-221f096eb55f-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.759560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.759492 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8pf\" (UniqueName: \"kubernetes.io/projected/5eb74ec9-129c-44bf-8b1d-221f096eb55f-kube-api-access-st8pf\") pod \"opendatahub-operator-controller-manager-55fb46c94b-l292v\" (UID: \"5eb74ec9-129c-44bf-8b1d-221f096eb55f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.764205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.764177 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:03.806509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.806475 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:03.997840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:03.997793 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:04.224084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:04.224054 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:04.305820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:04.305785 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v"] Nov 02 15:01:04.308456 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:04.308422 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb74ec9_129c_44bf_8b1d_221f096eb55f.slice/crio-32384809e73c90a108f7ece0d4e262002d65bea3f2adfc7b9bed0cdd294253a3 WatchSource:0}: Error finding container 32384809e73c90a108f7ece0d4e262002d65bea3f2adfc7b9bed0cdd294253a3: Status 404 returned error can't find the container with id 32384809e73c90a108f7ece0d4e262002d65bea3f2adfc7b9bed0cdd294253a3 Nov 02 15:01:04.839565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:04.839528 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:05.053818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:05.053760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:05.306655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:05.306618 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" event={"ID":"5eb74ec9-129c-44bf-8b1d-221f096eb55f","Type":"ContainerStarted","Data":"32384809e73c90a108f7ece0d4e262002d65bea3f2adfc7b9bed0cdd294253a3"} Nov 02 15:01:06.190590 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:01:06.190552 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:01:06.413877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:06.413837 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:06.631683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:06.631647 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:07.015640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:07.015608 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:07.565384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:07.565350 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:08.318508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:08.318477 4469 generic.go:356] "Generic (PLEG): container finished" podID="5eb74ec9-129c-44bf-8b1d-221f096eb55f" containerID="87e7c0da2f1020e468db51ad985d424c2198027de6460f6832fda81496255fd2" exitCode=0 Nov 02 15:01:08.318643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:08.318533 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" event={"ID":"5eb74ec9-129c-44bf-8b1d-221f096eb55f","Type":"ContainerDied","Data":"87e7c0da2f1020e468db51ad985d424c2198027de6460f6832fda81496255fd2"} Nov 02 15:01:08.537966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:08.537924 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:08.857522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:08.857453 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:09.323913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:09.323876 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" event={"ID":"5eb74ec9-129c-44bf-8b1d-221f096eb55f","Type":"ContainerStarted","Data":"2984bac35d35d5330391170eaece398d492f731cb978234d2a555491817ebee5"} Nov 02 15:01:09.324049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:09.323985 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:09.472950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:09.472901 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" podStartSLOduration=1.911825027 podStartE2EDuration="6.472887206s" totalImagesPullingTime="4.07488011s" totalInitContainerRuntime="486.182069ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:09.471030029 +0000 UTC m=+272.354632548" watchObservedRunningTime="2026-11-02 15:01:09.472887206 +0000 UTC m=+272.356489804" Nov 02 15:01:10.871505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:10.871473 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:11.088992 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:11.088952 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:11.306428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:11.306399 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:12.356538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:12.356504 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:12.581607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:12.581573 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:14.477483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:14.477450 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:14.983972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:14.983939 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:01:15.022823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.022150 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.043001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.042973 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:01:15.086604 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.086580 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.086733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.086617 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zs6\" (UniqueName: \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.091128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.091097 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 15:01:15.091229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.091188 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 15:01:15.091268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.091241 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-22tph\"" Nov 02 15:01:15.187280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.187240 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.187280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.187286 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zs6\" (UniqueName: \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.187634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.187615 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.248263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.248200 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zs6\" (UniqueName: \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\") pod \"kuadrant-operator-controller-manager-678d87766f-bnl8l\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.337632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.337597 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:15.834482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:15.834437 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:01:15.838661 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:15.838627 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b573780_88a8_4266_ae14_fdea1146f271.slice/crio-cc2755cc55c599ca700912acdd7c71d5c371ac7c52a1f65c3d06cc6e0c1b1890 WatchSource:0}: Error finding container cc2755cc55c599ca700912acdd7c71d5c371ac7c52a1f65c3d06cc6e0c1b1890: Status 404 returned error can't find the container with id cc2755cc55c599ca700912acdd7c71d5c371ac7c52a1f65c3d06cc6e0c1b1890 Nov 02 15:01:16.347791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:16.347756 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" event={"ID":"3b573780-88a8-4266-ae14-fdea1146f271","Type":"ContainerStarted","Data":"cc2755cc55c599ca700912acdd7c71d5c371ac7c52a1f65c3d06cc6e0c1b1890"} Nov 02 15:01:16.806941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:16.806903 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:17.073160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:17.073080 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:17.401665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:17.401588 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:17.604633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:17.604593 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:17.840491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:17.840451 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:20.329294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:20.329268 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-l292v" Nov 02 15:01:21.369377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:21.369335 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" event={"ID":"3b573780-88a8-4266-ae14-fdea1146f271","Type":"ContainerStarted","Data":"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb"} Nov 02 15:01:21.369744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:21.369475 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:21.492634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:21.492571 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" podStartSLOduration=2.067055277 podStartE2EDuration="7.492551212s" totalImagesPullingTime="5.425495935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:21.491434729 +0000 UTC m=+284.375037247" watchObservedRunningTime="2026-11-02 15:01:21.492551212 +0000 UTC m=+284.376153731" Nov 02 15:01:22.736305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:22.736261 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:23.026935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.026851 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:23.321254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.321167 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj"] Nov 02 15:01:23.323487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.323465 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.378165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.378143 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 15:01:23.427668 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.427641 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 15:01:23.427866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.427718 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qbgm7\"" Nov 02 15:01:23.428818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.428779 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj"] Nov 02 15:01:23.465259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.465230 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.465355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.465280 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pz8\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-kube-api-access-54pz8\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.566313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.566281 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pz8\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-kube-api-access-54pz8\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.566435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.566372 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.639843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.639766 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pz8\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-kube-api-access-54pz8\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.640915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.640892 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aba03b2-70ba-49cb-96ee-896527cd0d76-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lxpmj\" (UID: \"0aba03b2-70ba-49cb-96ee-896527cd0d76\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:23.644191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:23.644175 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:24.020748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:24.020709 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj"] Nov 02 15:01:24.023227 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:24.023200 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aba03b2_70ba_49cb_96ee_896527cd0d76.slice/crio-d8c77a8cb9cbef75e3d3bb1a89456f4e19bcff15187977aff58470deba337bb4 WatchSource:0}: Error finding container d8c77a8cb9cbef75e3d3bb1a89456f4e19bcff15187977aff58470deba337bb4: Status 404 returned error can't find the container with id d8c77a8cb9cbef75e3d3bb1a89456f4e19bcff15187977aff58470deba337bb4 Nov 02 15:01:24.308563 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:24.308485 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:24.380564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:24.380532 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" event={"ID":"0aba03b2-70ba-49cb-96ee-896527cd0d76","Type":"ContainerStarted","Data":"d8c77a8cb9cbef75e3d3bb1a89456f4e19bcff15187977aff58470deba337bb4"} Nov 02 15:01:25.248659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.248628 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:25.492759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.492724 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj"] Nov 02 15:01:25.495429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.495411 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.552340 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.552306 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cbn8f\"" Nov 02 15:01:25.582897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.582857 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.583049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.582918 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt8d\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-kube-api-access-jbt8d\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.601775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.601735 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj"] Nov 02 15:01:25.638265 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.638232 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:25.683629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.683604 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.683793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.683652 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt8d\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-kube-api-access-jbt8d\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.750303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.750274 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt8d\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-kube-api-access-jbt8d\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.750303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.750286 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca429e27-0085-43e5-a10b-e72e64e895e0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hxndj\" (UID: \"ca429e27-0085-43e5-a10b-e72e64e895e0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:25.807292 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:25.807204 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" Nov 02 15:01:26.135225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.135159 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d"] Nov 02 15:01:26.137499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.137477 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.192581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.192100 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 15:01:26.193421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.193398 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9nhl2\"" Nov 02 15:01:26.193814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.193777 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj"] Nov 02 15:01:26.194410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.194388 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:26.196813 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:26.196780 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca429e27_0085_43e5_a10b_e72e64e895e0.slice/crio-a1382ab589735f79a1a364f36328b2887d80fa0bfb25c0e4dde3d787bf29052a WatchSource:0}: Error finding container a1382ab589735f79a1a364f36328b2887d80fa0bfb25c0e4dde3d787bf29052a: Status 404 returned error can't find the container with id a1382ab589735f79a1a364f36328b2887d80fa0bfb25c0e4dde3d787bf29052a Nov 02 15:01:26.241351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.241324 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 15:01:26.242773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.242747 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d"] Nov 02 15:01:26.288267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.288236 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.288692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.288331 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.288692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.288399 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sms5\" (UniqueName: \"kubernetes.io/projected/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-kube-api-access-4sms5\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.389630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.389503 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.389630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.389594 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.389885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.389633 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sms5\" (UniqueName: \"kubernetes.io/projected/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-kube-api-access-4sms5\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.390663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.390625 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.391144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.391076 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" event={"ID":"ca429e27-0085-43e5-a10b-e72e64e895e0","Type":"ContainerStarted","Data":"a1382ab589735f79a1a364f36328b2887d80fa0bfb25c0e4dde3d787bf29052a"} Nov 02 15:01:26.392744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.392724 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.417550 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.417524 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:26.481670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.481639 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sms5\" (UniqueName: \"kubernetes.io/projected/b0a3e2cd-6a7f-498d-9b05-3451e1df5d29-kube-api-access-4sms5\") pod \"kuadrant-console-plugin-66c4fb575d-8qt7d\" (UID: \"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:26.665080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.665001 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:26.747731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:26.747694 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" Nov 02 15:01:27.167669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:27.167641 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d"] Nov 02 15:01:27.170342 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:27.170312 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a3e2cd_6a7f_498d_9b05_3451e1df5d29.slice/crio-431a5699e9e0e8fae7c2df626e7dd345ceafc8cd1d77f3947e91f8730806b132 WatchSource:0}: Error finding container 431a5699e9e0e8fae7c2df626e7dd345ceafc8cd1d77f3947e91f8730806b132: Status 404 returned error can't find the container with id 431a5699e9e0e8fae7c2df626e7dd345ceafc8cd1d77f3947e91f8730806b132 Nov 02 15:01:27.395642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:27.395610 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" event={"ID":"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29","Type":"ContainerStarted","Data":"431a5699e9e0e8fae7c2df626e7dd345ceafc8cd1d77f3947e91f8730806b132"} Nov 02 15:01:27.506212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:27.506173 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:27.719680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:27.719629 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:29.411075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:29.411035 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" event={"ID":"0aba03b2-70ba-49cb-96ee-896527cd0d76","Type":"ContainerStarted","Data":"53fff9698065cca2997373737a8dc90ec62628a5d4cd835b57fcf81f99f13f32"} Nov 02 15:01:29.411862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:29.411835 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:29.531483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:29.531409 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" podStartSLOduration=1.64241502 podStartE2EDuration="6.531388998s" totalImagesPullingTime="4.888973978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:29.530022085 +0000 UTC m=+292.413624605" watchObservedRunningTime="2026-11-02 15:01:29.531388998 +0000 UTC m=+292.414991517" Nov 02 15:01:30.417592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:30.417553 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" event={"ID":"ca429e27-0085-43e5-a10b-e72e64e895e0","Type":"ContainerStarted","Data":"1a01118d6a0c078347915ed400adce975d402f0bf7d306f116df0261de2fa845"} Nov 02 15:01:30.534023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:30.532894 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hxndj" podStartSLOduration=2.096756031 podStartE2EDuration="5.532872528s" totalImagesPullingTime="3.436116497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:30.5319098 +0000 UTC m=+293.415512322" watchObservedRunningTime="2026-11-02 15:01:30.532872528 +0000 UTC m=+293.416475048" Nov 02 15:01:31.360236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:31.360158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:31.576778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:31.576730 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:32.375947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:32.375919 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:01:32.395676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:32.395634 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:32.614879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:32.614838 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:32.832258 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:32.832165 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:33.105085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.105009 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:33.346908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.346881 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:33.433200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.433121 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" event={"ID":"b0a3e2cd-6a7f-498d-9b05-3451e1df5d29","Type":"ContainerStarted","Data":"6297a13e16bc956291c47cbdc2505f92806caee5354f3b1bf59c8d02ccb422c1"} Nov 02 15:01:33.582897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.582842 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8qt7d" podStartSLOduration=1.843097814 podStartE2EDuration="7.58282747s" totalImagesPullingTime="5.739729656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:33.581680236 +0000 UTC m=+296.465282755" watchObservedRunningTime="2026-11-02 15:01:33.58282747 +0000 UTC m=+296.466429988" Nov 02 15:01:33.635537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.635501 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:33.882577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:33.882537 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:36.185322 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:01:36.185276 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:01:36.425156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:36.425128 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" Nov 02 15:01:39.101085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:39.101045 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:39.307439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:39.307406 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:40.665655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:40.665623 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:40.870085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:40.870045 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:41.327452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:41.327420 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:42.746242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.746142 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:42.814129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.814075 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-q7dr4"] Nov 02 15:01:42.817899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.817869 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:42.870227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.870190 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 15:01:42.870388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.870232 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 15:01:42.870476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.870457 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 15:01:42.919969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.919943 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 15:01:42.919969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.919957 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qwfj6\"" Nov 02 15:01:42.921342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.921322 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-q7dr4"] Nov 02 15:01:42.939853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.939825 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tmp\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:42.939975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.939926 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5dd\" (UniqueName: \"kubernetes.io/projected/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-kube-api-access-js5dd\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:42.939975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.939963 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tls-certs\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:42.954030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:42.954006 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:43.041282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.041174 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tmp\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.041282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.041209 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5dd\" (UniqueName: \"kubernetes.io/projected/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-kube-api-access-js5dd\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.041282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.041235 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tls-certs\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.043511 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.043489 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tmp\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.043704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.043686 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-tls-certs\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.102228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.102204 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5dd\" (UniqueName: \"kubernetes.io/projected/d5f80fb2-d52b-4327-87dc-a24f1e6da3ae-kube-api-access-js5dd\") pod \"kube-auth-proxy-d9588f984-q7dr4\" (UID: \"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.131564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.131542 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" Nov 02 15:01:43.307084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.306985 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:43.504432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.504384 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-q7dr4"] Nov 02 15:01:43.507023 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:43.506995 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f80fb2_d52b_4327_87dc_a24f1e6da3ae.slice/crio-78409d0d95bd84afcb96c4feab35e8700b5507bbe5599f26485c20b6e2655dba WatchSource:0}: Error finding container 78409d0d95bd84afcb96c4feab35e8700b5507bbe5599f26485c20b6e2655dba: Status 404 returned error can't find the container with id 78409d0d95bd84afcb96c4feab35e8700b5507bbe5599f26485c20b6e2655dba Nov 02 15:01:43.508903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.508883 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:01:43.516067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.516020 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:43.843356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:43.843323 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:44.488367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:44.488328 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" event={"ID":"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae","Type":"ContainerStarted","Data":"78409d0d95bd84afcb96c4feab35e8700b5507bbe5599f26485c20b6e2655dba"} Nov 02 15:01:45.419823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:45.419771 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:45.760580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:45.760477 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:45.968508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:45.968465 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:47.143484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:47.143452 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:47.364994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:47.364956 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:47.503091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:47.503060 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" event={"ID":"d5f80fb2-d52b-4327-87dc-a24f1e6da3ae","Type":"ContainerStarted","Data":"2b32a807efcb22349b6e3fd757ef38948cb5300c0c4028ed3d3255a16061ff83"} Nov 02 15:01:47.621448 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:47.621387 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d9588f984-q7dr4" podStartSLOduration=2.564695461 podStartE2EDuration="5.621367251s" totalImagesPullingTime="3.05667179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:47.620298139 +0000 UTC m=+310.503900658" watchObservedRunningTime="2026-11-02 15:01:47.621367251 +0000 UTC m=+310.504969771" Nov 02 15:01:49.833915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.833841 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:01:49.836457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.836439 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:49.888649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.888621 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 15:01:49.907038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.907013 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-config-file\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:49.907145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.907046 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7l9\" (UniqueName: \"kubernetes.io/projected/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-kube-api-access-fr7l9\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:49.939088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:49.939062 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:01:50.008044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.008019 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-config-file\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:50.008183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.008051 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7l9\" (UniqueName: \"kubernetes.io/projected/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-kube-api-access-fr7l9\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:50.008640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.008622 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-config-file\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:50.069295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.069266 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7l9\" (UniqueName: \"kubernetes.io/projected/d38cccc7-83e2-4d4f-a826-7fbef4a61ac6-kube-api-access-fr7l9\") pod \"limitador-limitador-7c9b87d574-pvcb2\" (UID: \"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:50.139910 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.139839 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:01:50.147565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.147539 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:50.515102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.515074 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:01:50.517992 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:01:50.517958 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38cccc7_83e2_4d4f_a826_7fbef4a61ac6.slice/crio-3d758d32451b29a4cc589622b9a04a47f5a27043651adfa47043178780acd6fb WatchSource:0}: Error finding container 3d758d32451b29a4cc589622b9a04a47f5a27043651adfa47043178780acd6fb: Status 404 returned error can't find the container with id 3d758d32451b29a4cc589622b9a04a47f5a27043651adfa47043178780acd6fb Nov 02 15:01:50.768574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:50.768483 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:51.520250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:51.520215 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" event={"ID":"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6","Type":"ContainerStarted","Data":"3d758d32451b29a4cc589622b9a04a47f5a27043651adfa47043178780acd6fb"} Nov 02 15:01:52.289573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:52.289534 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:52.491555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:52.491496 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:52.709619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:52.709586 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:52.927793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:52.927760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:53.141678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:53.141593 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:53.530420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:53.530388 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" event={"ID":"d38cccc7-83e2-4d4f-a826-7fbef4a61ac6","Type":"ContainerStarted","Data":"4bc08a67c97b2c57ca4dbbe2600d00d9534c2b0576f08d4a02e35ba4265e1134"} Nov 02 15:01:53.530587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:53.530540 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:01:53.657323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:53.657233 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" podStartSLOduration=1.8559924799999998 podStartE2EDuration="4.657218069s" totalImagesPullingTime="2.801225589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:53.656463932 +0000 UTC m=+316.540066451" watchObservedRunningTime="2026-11-02 15:01:53.657218069 +0000 UTC m=+316.540820587" Nov 02 15:01:58.116129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:58.116091 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:58.319767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:58.319725 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:58.528257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:01:58.528223 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:04.535630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:04.535592 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pvcb2" Nov 02 15:02:06.184850 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:02:06.184811 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:02:08.289747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:08.289711 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:08.491296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:08.491263 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:09.167710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.167671 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:09.238146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.238115 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq"] Nov 02 15:02:09.256965 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.256937 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.303394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.303363 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq"] Nov 02 15:02:09.351514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.351486 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 15:02:09.385266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.385233 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.385402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.385275 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.385402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.385301 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0b1a77-b60d-4066-8bb4-2408402d3522-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.385518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.385402 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stgw\" (UniqueName: \"kubernetes.io/projected/af0b1a77-b60d-4066-8bb4-2408402d3522-kube-api-access-2stgw\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.401222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.401198 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 15:02:09.451144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.451079 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 15:02:09.451249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.451182 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-vwcwp\"" Nov 02 15:02:09.486441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486415 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0b1a77-b60d-4066-8bb4-2408402d3522-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.486602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486458 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stgw\" (UniqueName: \"kubernetes.io/projected/af0b1a77-b60d-4066-8bb4-2408402d3522-kube-api-access-2stgw\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.486602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486521 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.486602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486548 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.486960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486944 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.487006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.486979 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b1a77-b60d-4066-8bb4-2408402d3522-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.487339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.487322 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0b1a77-b60d-4066-8bb4-2408402d3522-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.547368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.547345 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stgw\" (UniqueName: \"kubernetes.io/projected/af0b1a77-b60d-4066-8bb4-2408402d3522-kube-api-access-2stgw\") pod \"kserve-module-controller-manager-7566cf5fc8-s9wlq\" (UID: \"af0b1a77-b60d-4066-8bb4-2408402d3522\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.568418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.568397 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:09.989427 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:09.989406 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq"] Nov 02 15:02:09.991832 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:02:09.991788 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0b1a77_b60d_4066_8bb4_2408402d3522.slice/crio-6de39ef6e1bb0357a2fb6db8e3ab4d2b2c3f768a17c58e815df505d81d708e7a WatchSource:0}: Error finding container 6de39ef6e1bb0357a2fb6db8e3ab4d2b2c3f768a17c58e815df505d81d708e7a: Status 404 returned error can't find the container with id 6de39ef6e1bb0357a2fb6db8e3ab4d2b2c3f768a17c58e815df505d81d708e7a Nov 02 15:02:10.599037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:10.599002 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" event={"ID":"af0b1a77-b60d-4066-8bb4-2408402d3522","Type":"ContainerStarted","Data":"6de39ef6e1bb0357a2fb6db8e3ab4d2b2c3f768a17c58e815df505d81d708e7a"} Nov 02 15:02:12.608978 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:12.608934 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" event={"ID":"af0b1a77-b60d-4066-8bb4-2408402d3522","Type":"ContainerStarted","Data":"3fdcdae7817de1e19a6e73c23b99ae9d19234436b6cc75e5685053970d4ec2d2"} Nov 02 15:02:12.609378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:12.609029 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:24.672567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:24.672535 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.155655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.155587 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" podStartSLOduration=19.267227769 podStartE2EDuration="21.155567055s" totalImagesPullingTime="1.888339286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:12.781865511 +0000 UTC m=+335.665468030" watchObservedRunningTime="2026-11-02 15:02:30.155567055 +0000 UTC m=+353.039169575" Nov 02 15:02:30.157158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.157130 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:30.160102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.160082 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.175008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.174984 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.212425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.212401 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:02:30.261636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.261609 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-77ggk\"" Nov 02 15:02:30.261788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.261722 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:02:30.262064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.262042 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:02:30.263437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.263412 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7c2\" (UniqueName: \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.263566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.263547 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.263620 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.263594 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.263676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.263614 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:30.364256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.364224 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.364407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.364270 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.364407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.364320 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7c2\" (UniqueName: \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.366515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.366491 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.366641 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.366623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.382004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.381977 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.426239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.426182 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7c2\" (UniqueName: \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\") pod \"maas-controller-579cf7c6d-gfrph\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.471256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.471230 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:30.587753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.587710 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.841198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:30.841167 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:30.843863 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:02:30.843827 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066422e3_d72d_42c1_89d5_3332adf12f26.slice/crio-e3922d62c46214d24fdb1fe945d4b9a1d04b7879c77c4c07f943f3e57fe7345d WatchSource:0}: Error finding container e3922d62c46214d24fdb1fe945d4b9a1d04b7879c77c4c07f943f3e57fe7345d: Status 404 returned error can't find the container with id e3922d62c46214d24fdb1fe945d4b9a1d04b7879c77c4c07f943f3e57fe7345d Nov 02 15:02:31.689233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:31.689191 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gfrph" event={"ID":"066422e3-d72d-42c1-89d5-3332adf12f26","Type":"ContainerStarted","Data":"e3922d62c46214d24fdb1fe945d4b9a1d04b7879c77c4c07f943f3e57fe7345d"} Nov 02 15:02:33.613889 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:33.613860 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-s9wlq" Nov 02 15:02:33.948217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:33.948140 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:36.191292 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:02:36.191253 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:02:36.661712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:36.661678 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:36.716691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:36.716658 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gfrph" event={"ID":"066422e3-d72d-42c1-89d5-3332adf12f26","Type":"ContainerStarted","Data":"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef"} Nov 02 15:02:36.716859 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:36.716744 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:36.835788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:36.835730 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-gfrph" podStartSLOduration=1.458703372 podStartE2EDuration="6.835715226s" totalImagesPullingTime="5.377011854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:36.835014919 +0000 UTC m=+359.718617439" watchObservedRunningTime="2026-11-02 15:02:36.835715226 +0000 UTC m=+359.719317745" Nov 02 15:02:36.872613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:36.872581 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:39.827312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:39.827278 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:40.133546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:40.133444 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:47.722321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:47.722284 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:48.410540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.410503 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:48.410728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.410703 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-gfrph" podUID="066422e3-d72d-42c1-89d5-3332adf12f26" containerName="manager" containerID="cri-o://e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef" gracePeriod=30 Nov 02 15:02:48.644815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.644770 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:48.722844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.722817 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\") pod \"066422e3-d72d-42c1-89d5-3332adf12f26\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " Nov 02 15:02:48.723255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.722953 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\" (UniqueName: \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\") pod \"066422e3-d72d-42c1-89d5-3332adf12f26\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " Nov 02 15:02:48.723255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.723005 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\") pod \"066422e3-d72d-42c1-89d5-3332adf12f26\" (UID: \"066422e3-d72d-42c1-89d5-3332adf12f26\") " Nov 02 15:02:48.725122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.725093 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2" pod "066422e3-d72d-42c1-89d5-3332adf12f26" (UID: "066422e3-d72d-42c1-89d5-3332adf12f26"). InnerVolumeSpecName "kube-api-access-4v7c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:02:48.725241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.725209 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs" pod "066422e3-d72d-42c1-89d5-3332adf12f26" (UID: "066422e3-d72d-42c1-89d5-3332adf12f26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:02:48.725906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.725883 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert" pod "066422e3-d72d-42c1-89d5-3332adf12f26" (UID: "066422e3-d72d-42c1-89d5-3332adf12f26"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:02:48.769450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.769412 4469 generic.go:356] "Generic (PLEG): container finished" podID="066422e3-d72d-42c1-89d5-3332adf12f26" containerID="e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef" exitCode=0 Nov 02 15:02:48.769630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.769480 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gfrph" Nov 02 15:02:48.769630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.769489 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gfrph" event={"ID":"066422e3-d72d-42c1-89d5-3332adf12f26","Type":"ContainerDied","Data":"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef"} Nov 02 15:02:48.769630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.769570 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gfrph" event={"ID":"066422e3-d72d-42c1-89d5-3332adf12f26","Type":"ContainerDied","Data":"e3922d62c46214d24fdb1fe945d4b9a1d04b7879c77c4c07f943f3e57fe7345d"} Nov 02 15:02:48.769630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.769602 4469 scope.go:122] "RemoveContainer" containerID="e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef" Nov 02 15:02:48.779029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.779012 4469 scope.go:122] "RemoveContainer" containerID="e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef" Nov 02 15:02:48.779313 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:02:48.779289 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef\": container with ID starting with e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef not found: ID does not exist" containerID="e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef" Nov 02 15:02:48.779365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.779329 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef"} err="failed to get container status \"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef\": rpc error: code = NotFound desc = could not find container \"e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef\": container with ID starting with e8555ce1f2cf666feb9c6f61cc6fe192bc89a3ebb3a2cddd9f16d126d9b7e9ef not found: ID does not exist" Nov 02 15:02:48.824372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.824346 4469 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-webhook-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:02:48.824372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.824367 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/066422e3-d72d-42c1-89d5-3332adf12f26-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:02:48.824490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.824379 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7c2\" (UniqueName: \"kubernetes.io/projected/066422e3-d72d-42c1-89d5-3332adf12f26-kube-api-access-4v7c2\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:02:48.933569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.933531 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:48.937856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:48.937828 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gfrph"] Nov 02 15:02:49.169034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:49.168947 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:49.236450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:02:49.236418 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066422e3-d72d-42c1-89d5-3332adf12f26" path="/var/lib/kubelet/pods/066422e3-d72d-42c1-89d5-3332adf12f26/volumes" Nov 02 15:03:06.184651 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:03:06.184608 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:03:10.084562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:10.084527 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:10.298999 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:10.298962 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:12.280657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:12.280624 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:12.496893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:12.496854 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:12.745941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:12.745905 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:13.290058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.290020 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:13.290450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.290344 4469 cpu_manager.go:414] "containerMap: removing container" podUID="066422e3-d72d-42c1-89d5-3332adf12f26" containerName="manager" Nov 02 15:03:13.290450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.290354 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066422e3-d72d-42c1-89d5-3332adf12f26" containerName="manager" Nov 02 15:03:13.290450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.290403 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="066422e3-d72d-42c1-89d5-3332adf12f26" containerName="manager" Nov 02 15:03:13.295958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.295941 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.398000 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.397972 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 15:03:13.398168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.398102 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vkjdq\"" Nov 02 15:03:13.398168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.398109 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:03:13.398327 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.398191 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:03:13.398327 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.398278 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:03:13.399737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.399719 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:13.433716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.433689 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.433871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.433730 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td528\" (UniqueName: \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.433871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.433793 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.433871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.433835 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.534915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.534887 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.534915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.534916 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.535131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.534967 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.535131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.534987 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td528\" (UniqueName: \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.535640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.535611 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.537477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.537456 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.537559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.537456 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.595580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.595522 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td528\" (UniqueName: \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\") pod \"maas-api-6d669c7fcd-ggg9f\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.607384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.607365 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:13.975634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:13.975603 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:13.978974 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:03:13.978950 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e03ed25_2ba2_4460_979f_c31c171a1314.slice/crio-44249dafc00929b560a61d2c103f65d1c33d5fbd9302bd611aacd13c60642ffb WatchSource:0}: Error finding container 44249dafc00929b560a61d2c103f65d1c33d5fbd9302bd611aacd13c60642ffb: Status 404 returned error can't find the container with id 44249dafc00929b560a61d2c103f65d1c33d5fbd9302bd611aacd13c60642ffb Nov 02 15:03:14.885923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:14.885884 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" event={"ID":"5e03ed25-2ba2-4460-979f-c31c171a1314","Type":"ContainerStarted","Data":"44249dafc00929b560a61d2c103f65d1c33d5fbd9302bd611aacd13c60642ffb"} Nov 02 15:03:14.931291 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:14.931251 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:15.147957 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:15.147884 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:15.360299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:15.360268 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:15.569273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:15.569231 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:15.990794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:15.990726 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h92jq"] Nov 02 15:03:15.994074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:15.994050 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.087613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.087535 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:16.096178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.096148 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 15:03:16.096178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.096149 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-q9v4f\"" Nov 02 15:03:16.097713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.097687 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h92jq"] Nov 02 15:03:16.159504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.159469 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhd7\" (UniqueName: \"kubernetes.io/projected/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-kube-api-access-tbhd7\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.159655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.159600 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-tls-certs\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.261326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.261300 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-tls-certs\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.261445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.261355 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhd7\" (UniqueName: \"kubernetes.io/projected/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-kube-api-access-tbhd7\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.263667 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.263645 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-tls-certs\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.293878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.293850 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:16.322617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.322592 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhd7\" (UniqueName: \"kubernetes.io/projected/1cb6d8c6-fde1-4eee-b43e-0efb1c855907-kube-api-access-tbhd7\") pod \"model-serving-api-75bd89fdfc-h92jq\" (UID: \"1cb6d8c6-fde1-4eee-b43e-0efb1c855907\") " pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.607908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.607796 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:16.896751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.896665 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" event={"ID":"5e03ed25-2ba2-4460-979f-c31c171a1314","Type":"ContainerStarted","Data":"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57"} Nov 02 15:03:16.896927 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.896827 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:16.976837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:16.976770 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h92jq"] Nov 02 15:03:16.979431 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:03:16.979392 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb6d8c6_fde1_4eee_b43e_0efb1c855907.slice/crio-82015b9cbefe0690a40252052085ffdcf8d2afbccd6dedf2aae876b0001a8abb WatchSource:0}: Error finding container 82015b9cbefe0690a40252052085ffdcf8d2afbccd6dedf2aae876b0001a8abb: Status 404 returned error can't find the container with id 82015b9cbefe0690a40252052085ffdcf8d2afbccd6dedf2aae876b0001a8abb Nov 02 15:03:17.014873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:17.014788 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" podStartSLOduration=1.737900345 podStartE2EDuration="4.014768331s" totalImagesPullingTime="2.276867986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:17.01374759 +0000 UTC m=+399.897350109" watchObservedRunningTime="2026-11-02 15:03:17.014768331 +0000 UTC m=+399.898370851" Nov 02 15:03:17.202690 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:17.202621 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:17.902064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:17.902030 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" event={"ID":"1cb6d8c6-fde1-4eee-b43e-0efb1c855907","Type":"ContainerStarted","Data":"82015b9cbefe0690a40252052085ffdcf8d2afbccd6dedf2aae876b0001a8abb"} Nov 02 15:03:18.865354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:18.865319 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:03:19.912211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:19.912174 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" event={"ID":"1cb6d8c6-fde1-4eee-b43e-0efb1c855907","Type":"ContainerStarted","Data":"20597c22e23b1cadaf38a6c74ea2e35418e55660d9eaa001371c10b6507118cb"} Nov 02 15:03:19.912678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:19.912298 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:20.029311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:20.029261 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" podStartSLOduration=2.6135174389999998 podStartE2EDuration="5.02924193s" totalImagesPullingTime="2.415724491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:20.029164443 +0000 UTC m=+402.912766954" watchObservedRunningTime="2026-11-02 15:03:20.02924193 +0000 UTC m=+402.912844449" Nov 02 15:03:22.345104 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:22.345067 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:22.551888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:22.551848 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:22.760027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:22.759993 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:22.908214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:22.908186 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:22.970056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:22.970021 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:23.183849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:23.183745 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:23.402674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:23.402640 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:23.616305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:23.616276 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:23.845887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:23.845856 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:25.032252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:25.032220 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:30.920217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:30.920183 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-h92jq" Nov 02 15:03:33.648361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.648326 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:33.648877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.648591 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" podUID="5e03ed25-2ba2-4460-979f-c31c171a1314" containerName="maas-api" containerID="cri-o://f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57" gracePeriod=30 Nov 02 15:03:33.893853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.893824 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:33.913977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.913900 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\") pod \"5e03ed25-2ba2-4460-979f-c31c171a1314\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " Nov 02 15:03:33.913977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.913975 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\" (UniqueName: \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\") pod \"5e03ed25-2ba2-4460-979f-c31c171a1314\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " Nov 02 15:03:33.916121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.916093 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs" pod "5e03ed25-2ba2-4460-979f-c31c171a1314" (UID: "5e03ed25-2ba2-4460-979f-c31c171a1314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:03:33.916250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.916234 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528" pod "5e03ed25-2ba2-4460-979f-c31c171a1314" (UID: "5e03ed25-2ba2-4460-979f-c31c171a1314"). InnerVolumeSpecName "kube-api-access-td528". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:03:33.968904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.968869 4469 generic.go:356] "Generic (PLEG): container finished" podID="5e03ed25-2ba2-4460-979f-c31c171a1314" containerID="f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57" exitCode=0 Nov 02 15:03:33.969086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.968919 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" Nov 02 15:03:33.969086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.968939 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" event={"ID":"5e03ed25-2ba2-4460-979f-c31c171a1314","Type":"ContainerDied","Data":"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57"} Nov 02 15:03:33.969086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.968992 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f" event={"ID":"5e03ed25-2ba2-4460-979f-c31c171a1314","Type":"ContainerDied","Data":"44249dafc00929b560a61d2c103f65d1c33d5fbd9302bd611aacd13c60642ffb"} Nov 02 15:03:33.969086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.969016 4469 scope.go:122] "RemoveContainer" containerID="f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57" Nov 02 15:03:33.981105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.981077 4469 scope.go:122] "RemoveContainer" containerID="f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57" Nov 02 15:03:33.981475 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:03:33.981451 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57\": container with ID starting with f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57 not found: ID does not exist" containerID="f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57" Nov 02 15:03:33.981543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:33.981488 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57"} err="failed to get container status \"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57\": rpc error: code = NotFound desc = could not find container \"f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57\": container with ID starting with f27cc9003f31d71808916233f8587e019c7ff30c099a3ebc9d94f39207dfcd57 not found: ID does not exist" Nov 02 15:03:34.014388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.014342 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\") pod \"5e03ed25-2ba2-4460-979f-c31c171a1314\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " Nov 02 15:03:34.014547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.014394 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\") pod \"5e03ed25-2ba2-4460-979f-c31c171a1314\" (UID: \"5e03ed25-2ba2-4460-979f-c31c171a1314\") " Nov 02 15:03:34.014717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.014696 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td528\" (UniqueName: \"kubernetes.io/projected/5e03ed25-2ba2-4460-979f-c31c171a1314-kube-api-access-td528\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:03:34.014756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.014727 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:03:34.014837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.014790 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca" pod "5e03ed25-2ba2-4460-979f-c31c171a1314" (UID: "5e03ed25-2ba2-4460-979f-c31c171a1314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:03:34.016860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.016831 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls" pod "5e03ed25-2ba2-4460-979f-c31c171a1314" (UID: "5e03ed25-2ba2-4460-979f-c31c171a1314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:03:34.115517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.115479 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e03ed25-2ba2-4460-979f-c31c171a1314-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:03:34.115517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.115512 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e03ed25-2ba2-4460-979f-c31c171a1314-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:03:34.299512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.299475 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:34.387832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.387782 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:34.392982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:34.392941 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ggg9f"] Nov 02 15:03:35.237385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:03:35.237351 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e03ed25-2ba2-4460-979f-c31c171a1314" path="/var/lib/kubelet/pods/5e03ed25-2ba2-4460-979f-c31c171a1314/volumes" Nov 02 15:03:36.186976 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:03:36.186935 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:04:06.185880 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:04:06.185837 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:04:25.358759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:25.358723 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:34.638664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:34.638616 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:35.526633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:35.526596 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:36.184874 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:04:36.184840 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:04:53.572510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:53.572476 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:55.825969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:55.825936 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:56.029131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:04:56.029092 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:03.791587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:03.791551 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:05.256479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:05.256449 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:05.481384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:05.481353 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:06.185341 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:05:06.185305 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:05:22.718620 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.718573 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm"] Nov 02 15:05:22.719148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.719123 4469 cpu_manager.go:414] "containerMap: removing container" podUID="5e03ed25-2ba2-4460-979f-c31c171a1314" containerName="maas-api" Nov 02 15:05:22.719148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.719144 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e03ed25-2ba2-4460-979f-c31c171a1314" containerName="maas-api" Nov 02 15:05:22.719261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.719229 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e03ed25-2ba2-4460-979f-c31c171a1314" containerName="maas-api" Nov 02 15:05:22.722157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.722134 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746107 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746154 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746254 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746306 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746411 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.746490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.746444 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9sgs\" (UniqueName: \"kubernetes.io/projected/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-kube-api-access-f9sgs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.787029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.786993 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 15:05:22.800844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.800792 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 15:05:22.826225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.826188 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vgzsg\"" Nov 02 15:05:22.826225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.826198 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 15:05:22.829265 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.829223 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm"] Nov 02 15:05:22.847838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.847781 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.847849 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9sgs\" (UniqueName: \"kubernetes.io/projected/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-kube-api-access-f9sgs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.847899 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.847925 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.847994 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.848056 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.848355 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.848373 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.848500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.848429 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.850347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.850323 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.850516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.850496 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:22.877146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.877107 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:22.909401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:22.909362 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9sgs\" (UniqueName: \"kubernetes.io/projected/bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e-kube-api-access-f9sgs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm\" (UID: \"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:23.035249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:23.035151 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:23.410985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:23.410901 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm"] Nov 02 15:05:23.414049 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:05:23.414020 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8c5e76_d6ba_48b8_a50b_9e19d4c4860e.slice/crio-4fcd385f5e4a290d02f9a0227bd12bd8221d6ad10a9f1f08c5980d407f4655c6 WatchSource:0}: Error finding container 4fcd385f5e4a290d02f9a0227bd12bd8221d6ad10a9f1f08c5980d407f4655c6: Status 404 returned error can't find the container with id 4fcd385f5e4a290d02f9a0227bd12bd8221d6ad10a9f1f08c5980d407f4655c6 Nov 02 15:05:23.437261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:23.437220 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" event={"ID":"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e","Type":"ContainerStarted","Data":"4fcd385f5e4a290d02f9a0227bd12bd8221d6ad10a9f1f08c5980d407f4655c6"} Nov 02 15:05:25.706106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:25.706056 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:26.250773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:26.250732 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:05:26.454172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:26.454136 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" event={"ID":"bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e","Type":"ContainerStarted","Data":"3ac42e4307db1658c1addb9a90c0566b66d44b0376ddfc6e97a78f01a12d4fcd"} Nov 02 15:05:26.557972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:26.557873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:26.577583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:26.577529 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" podStartSLOduration=2.483145628 podStartE2EDuration="4.577511803s" totalImagesPullingTime="2.094366175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:26.576406482 +0000 UTC m=+529.460009002" watchObservedRunningTime="2026-11-02 15:05:26.577511803 +0000 UTC m=+529.461114356" Nov 02 15:05:27.036203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:27.036157 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:30.963794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:30.963747 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:33.035726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:33.035681 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:33.048977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:33.048937 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:33.496690 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:33.496659 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm" Nov 02 15:05:34.979184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:34.979132 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:35.148732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:35.148698 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:05:35.924947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:35.924908 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:36.185683 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:05:36.185592 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:05:36.609509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:36.609474 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:36.848251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:36.848214 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:39.260844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.260796 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb"] Nov 02 15:05:39.264263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.264244 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.316546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.316524 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 15:05:39.324895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.324868 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb"] Nov 02 15:05:39.396101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396070 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.396262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396105 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.396262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396125 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f791a3-de1e-4890-943a-ea5230fe6268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.396353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396284 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbvz\" (UniqueName: \"kubernetes.io/projected/25f791a3-de1e-4890-943a-ea5230fe6268-kube-api-access-shbvz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.396353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396326 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.396434 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.396376 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.412743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.412721 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:39.496815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.496771 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.496957 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.496916 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.496957 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.496939 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f791a3-de1e-4890-943a-ea5230fe6268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.496999 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbvz\" (UniqueName: \"kubernetes.io/projected/25f791a3-de1e-4890-943a-ea5230fe6268-kube-api-access-shbvz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.497029 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.497080 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.497227 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.497285 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.497447 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.497427 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.499398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.499369 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f791a3-de1e-4890-943a-ea5230fe6268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.499734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.499713 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f791a3-de1e-4890-943a-ea5230fe6268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.557724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.557665 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbvz\" (UniqueName: \"kubernetes.io/projected/25f791a3-de1e-4890-943a-ea5230fe6268-kube-api-access-shbvz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb\" (UID: \"25f791a3-de1e-4890-943a-ea5230fe6268\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.575540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.575510 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:39.945152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:39.945127 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb"] Nov 02 15:05:39.947179 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:05:39.947153 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25f791a3_de1e_4890_943a_ea5230fe6268.slice/crio-a2765a87a572d6c688b1dbbd4a113eeaf2c3f5cf2071ad8629b25c59f7b7c2d1 WatchSource:0}: Error finding container a2765a87a572d6c688b1dbbd4a113eeaf2c3f5cf2071ad8629b25c59f7b7c2d1: Status 404 returned error can't find the container with id a2765a87a572d6c688b1dbbd4a113eeaf2c3f5cf2071ad8629b25c59f7b7c2d1 Nov 02 15:05:40.514870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:40.514838 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" event={"ID":"25f791a3-de1e-4890-943a-ea5230fe6268","Type":"ContainerStarted","Data":"a2765a87a572d6c688b1dbbd4a113eeaf2c3f5cf2071ad8629b25c59f7b7c2d1"} Nov 02 15:05:41.526049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:41.526009 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" event={"ID":"25f791a3-de1e-4890-943a-ea5230fe6268","Type":"ContainerStarted","Data":"ba4291a674429ab827c596cff818587b133c66dfede894671ca6ac698a805974"} Nov 02 15:05:41.576458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:41.576429 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:41.643574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:41.643520 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podStartSLOduration=1.997398187 podStartE2EDuration="2.643506078s" totalImagesPullingTime="646.107891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:41.643119237 +0000 UTC m=+544.526721817" watchObservedRunningTime="2026-11-02 15:05:41.643506078 +0000 UTC m=+544.527108612" Nov 02 15:05:43.662695 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:43.662661 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:05:43.665823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:43.665781 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:48.547553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.547514 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:05:48.551188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.551166 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.596764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.596726 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:05:48.600214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.600190 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.603013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.602985 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6x9j8\"" Nov 02 15:05:48.644678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.644651 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 15:05:48.680038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680001 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680040 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgs7h\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680069 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680159 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680189 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680211 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680271 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680311 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680333 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680369 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680392 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680469 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680499 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680521 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680548 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680579 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.680786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680637 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.680786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.680709 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.694306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.694264 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:05:48.760400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.760359 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:05:48.781252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781216 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.781252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781251 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781271 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781292 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781325 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781349 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781379 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781418 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781452 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781477 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgs7h\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.781536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781502 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781555 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781576 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781601 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781633 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781671 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781700 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781757 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.781841 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.782050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782001 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782091 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782129 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782414 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782503 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782765 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.783027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.782827 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.783339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.783070 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.783339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.783194 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.784613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.784553 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.784854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.784775 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.784854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.784785 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.785616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.785586 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.843040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.842966 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.843040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.842966 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.843210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.843096 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6rhn6\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:48.843210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.843129 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgs7h\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9znhx\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.866784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.866745 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:48.913231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:48.913186 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:49.178050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.177946 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:49.295451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.295421 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:05:49.298694 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:05:49.298666 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5216990_fcd6_4b0c_8689_7c67e677d6b2.slice/crio-ab9c84dc8c72752be1900a84eef2b573038bf9416e62009a58fffad3a4ce2793 WatchSource:0}: Error finding container ab9c84dc8c72752be1900a84eef2b573038bf9416e62009a58fffad3a4ce2793: Status 404 returned error can't find the container with id ab9c84dc8c72752be1900a84eef2b573038bf9416e62009a58fffad3a4ce2793 Nov 02 15:05:49.345299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.345117 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:05:49.348138 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:05:49.348108 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6609b39_4ce0_4589_9363_34d0618e9be0.slice/crio-adeaab528069662395f601419fda77dfaba16eef24f233cd81afb62f4da89a8f WatchSource:0}: Error finding container adeaab528069662395f601419fda77dfaba16eef24f233cd81afb62f4da89a8f: Status 404 returned error can't find the container with id adeaab528069662395f601419fda77dfaba16eef24f233cd81afb62f4da89a8f Nov 02 15:05:49.561537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.561495 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" event={"ID":"f6609b39-4ce0-4589-9363-34d0618e9be0","Type":"ContainerStarted","Data":"adeaab528069662395f601419fda77dfaba16eef24f233cd81afb62f4da89a8f"} Nov 02 15:05:49.562796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.562771 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" event={"ID":"a5216990-fcd6-4b0c-8689-7c67e677d6b2","Type":"ContainerStarted","Data":"ab9c84dc8c72752be1900a84eef2b573038bf9416e62009a58fffad3a4ce2793"} Nov 02 15:05:49.575687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.575645 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:49.588042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:49.588015 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:50.578661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:50.578629 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:05:53.849494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:53.849455 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:05:53.984946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:53.984904 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:53.985090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:53.985006 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:53.985090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:53.985050 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:54.055932 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.055901 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4"] Nov 02 15:05:54.061240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.061208 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.116238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.116149 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 15:05:54.138657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.138617 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:54.138847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.138699 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:54.138847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.138728 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:05:54.152583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.152545 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4"] Nov 02 15:05:54.219956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.219920 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:54.242199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242153 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bsc\" (UniqueName: \"kubernetes.io/projected/fff589f3-5f1a-4549-8dcb-fa70077561c6-kube-api-access-64bsc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.242393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242210 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.242393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242366 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.242514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242416 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.242514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242472 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.242719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.242695 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff589f3-5f1a-4549-8dcb-fa70077561c6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.343638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343592 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff589f3-5f1a-4549-8dcb-fa70077561c6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.343880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343644 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bsc\" (UniqueName: \"kubernetes.io/projected/fff589f3-5f1a-4549-8dcb-fa70077561c6-kube-api-access-64bsc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.343880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343671 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.343880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343724 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.343880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343763 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.344091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.343871 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.344199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.344171 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.344549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.344221 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.344549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.344287 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.346005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.345985 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff589f3-5f1a-4549-8dcb-fa70077561c6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.346298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.346277 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff589f3-5f1a-4549-8dcb-fa70077561c6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.410062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.409990 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bsc\" (UniqueName: \"kubernetes.io/projected/fff589f3-5f1a-4549-8dcb-fa70077561c6-kube-api-access-64bsc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fpcc4\" (UID: \"fff589f3-5f1a-4549-8dcb-fa70077561c6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.592058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.592016 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" event={"ID":"a5216990-fcd6-4b0c-8689-7c67e677d6b2","Type":"ContainerStarted","Data":"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295"} Nov 02 15:05:54.593788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.593758 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" event={"ID":"f6609b39-4ce0-4589-9363-34d0618e9be0","Type":"ContainerStarted","Data":"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952"} Nov 02 15:05:54.680184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.680030 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:54.722098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.722046 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podStartSLOduration=1.884811042 podStartE2EDuration="6.722023475s" totalImagesPullingTime="4.837212433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.720728707 +0000 UTC m=+557.604331229" watchObservedRunningTime="2026-11-02 15:05:54.722023475 +0000 UTC m=+557.605626003" Nov 02 15:05:54.873884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.870062 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podStartSLOduration=2.236281342 podStartE2EDuration="6.870042341s" totalImagesPullingTime="4.633760999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.868290572 +0000 UTC m=+557.751893116" watchObservedRunningTime="2026-11-02 15:05:54.870042341 +0000 UTC m=+557.753644874" Nov 02 15:05:54.873884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.871038 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:05:54.913896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:54.913861 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:05:55.103791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.103749 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4"] Nov 02 15:05:55.110648 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:05:55.110621 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff589f3_5f1a_4549_8dcb_fa70077561c6.slice/crio-e0c56ed1043fa38f6afd7e36b8b5cf75f6fc764519acee49f4eb4d7dd5899650 WatchSource:0}: Error finding container e0c56ed1043fa38f6afd7e36b8b5cf75f6fc764519acee49f4eb4d7dd5899650: Status 404 returned error can't find the container with id e0c56ed1043fa38f6afd7e36b8b5cf75f6fc764519acee49f4eb4d7dd5899650 Nov 02 15:05:55.598875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.598832 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" event={"ID":"fff589f3-5f1a-4549-8dcb-fa70077561c6","Type":"ContainerStarted","Data":"e0c56ed1043fa38f6afd7e36b8b5cf75f6fc764519acee49f4eb4d7dd5899650"} Nov 02 15:05:55.868771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.868423 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:05:55.868771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.868479 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:05:55.914636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.914582 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:05:55.914998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:55.914633 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:05:56.605455 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.605410 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" event={"ID":"fff589f3-5f1a-4549-8dcb-fa70077561c6","Type":"ContainerStarted","Data":"3b842b3ee3c5a0832a8767d971c54e25bad5138dbca2afbe7ab94ffdc545c0cf"} Nov 02 15:05:56.680280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.680208 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:05:56.728448 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.728387 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" podStartSLOduration=2.042127051 podStartE2EDuration="2.728369136s" totalImagesPullingTime="686.242085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:56.728216974 +0000 UTC m=+559.611819494" watchObservedRunningTime="2026-11-02 15:05:56.728369136 +0000 UTC m=+559.611971656" Nov 02 15:05:56.868674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.868180 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:05:56.868674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.868243 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:05:56.913545 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.913507 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:05:56.913682 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:56.913559 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:05:57.869825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:57.869194 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:05:57.869825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:57.869260 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:05:57.914384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:57.914349 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:05:57.914517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:57.914397 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:05:58.878097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:58.877902 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:05:58.878097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:58.877971 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:05:58.914394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:58.914352 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:05:58.914545 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:58.914415 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:05:58.947825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:58.935308 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:59.874316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:59.874269 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:05:59.876365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:59.874335 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:05:59.914313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:59.914284 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:05:59.914739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:05:59.914709 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:06:00.867962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:00.867919 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9znhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 15:06:00.868174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:00.867988 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 15:06:00.914315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:00.914276 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 02 15:06:00.914760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:00.914358 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 02 15:06:01.966503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:01.966462 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:06:01.967009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:01.966837 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:06:01.967740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:01.967715 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:06:02.954966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:02.954929 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:06:02.955170 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:02.954990 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:06:03.913610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:03.913567 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:06:03.984529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:03.984489 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:06:03.984707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:03.984548 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:06:03.987717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:03.987693 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:06:04.655355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:04.655322 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:06:04.680703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:04.680679 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:06:04.692979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:04.692951 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:06:05.140330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:05.140295 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:06:05.679758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:05.679722 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fpcc4" Nov 02 15:06:06.186300 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:06:06.186258 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:06:13.981417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:13.981385 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:16.463740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:16.463699 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:18.854712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.854673 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:06:18.858474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.858451 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916187 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916233 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916257 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916389 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916418 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2x5\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916501 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916441 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916536 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916604 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916559 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.916604 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.916585 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:18.967145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:18.967113 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:06:19.017574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017543 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.017716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017581 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.017716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017616 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.017716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017645 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2x5\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.017716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017679 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.017716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017714 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017740 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017766 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017849 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.017999 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.018026 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.018146 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.018276 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.018569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.018535 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.020054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.020033 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.020281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.020260 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.078270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.078248 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.078359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.078276 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2x5\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7n2bx\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.171168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.171103 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:19.594919 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.594894 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:06:19.597358 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:06:19.597318 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468a249a_d376_4d44_97a5_e53286bfe366.slice/crio-14177254859df6ea544aac238f4ab66e1b6eab018c708afbcdfbba0f57aec7ca WatchSource:0}: Error finding container 14177254859df6ea544aac238f4ab66e1b6eab018c708afbcdfbba0f57aec7ca: Status 404 returned error can't find the container with id 14177254859df6ea544aac238f4ab66e1b6eab018c708afbcdfbba0f57aec7ca Nov 02 15:06:19.599491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.599459 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:06:19.599548 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.599523 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:06:19.599595 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.599552 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:06:19.727332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.727295 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" event={"ID":"468a249a-d376-4d44-97a5-e53286bfe366","Type":"ContainerStarted","Data":"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712"} Nov 02 15:06:19.727536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.727353 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" event={"ID":"468a249a-d376-4d44-97a5-e53286bfe366","Type":"ContainerStarted","Data":"14177254859df6ea544aac238f4ab66e1b6eab018c708afbcdfbba0f57aec7ca"} Nov 02 15:06:19.850872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:19.850741 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podStartSLOduration=1.850726388 podStartE2EDuration="1.850726388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:19.848236819 +0000 UTC m=+582.731839339" watchObservedRunningTime="2026-11-02 15:06:19.850726388 +0000 UTC m=+582.734328907" Nov 02 15:06:20.172206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:20.172120 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:20.173685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:20.173661 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:20.173757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:20.173715 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:21.172049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:21.172007 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:21.172248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:21.172059 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:22.172474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:22.172440 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:22.172940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:22.172517 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:23.172352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.172311 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:23.172530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.172373 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:23.440319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.440233 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:23.647212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.647177 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:23.844453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.844409 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-74wfk"] Nov 02 15:06:23.852039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.852007 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:23.890562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.890530 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-74wfk"] Nov 02 15:06:23.952824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.947083 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 15:06:23.952824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.947324 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-68tbq\"" Nov 02 15:06:23.962546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.962520 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j4tg5"] Nov 02 15:06:23.965939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.965916 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:23.985377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.985348 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d20065-2c84-4fbf-8cca-39aaf998c068-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:23.985528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:23.985506 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qptv\" (UniqueName: \"kubernetes.io/projected/e8d20065-2c84-4fbf-8cca-39aaf998c068-kube-api-access-2qptv\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.007504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.007064 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j4tg5"] Nov 02 15:06:24.086748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.086709 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.086922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.086765 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d20065-2c84-4fbf-8cca-39aaf998c068-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.086922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.086837 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9vs\" (UniqueName: \"kubernetes.io/projected/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-kube-api-access-vc9vs\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.087002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.086931 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qptv\" (UniqueName: \"kubernetes.io/projected/e8d20065-2c84-4fbf-8cca-39aaf998c068-kube-api-access-2qptv\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.087592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.087572 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d20065-2c84-4fbf-8cca-39aaf998c068-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.151730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.151645 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qptv\" (UniqueName: \"kubernetes.io/projected/e8d20065-2c84-4fbf-8cca-39aaf998c068-kube-api-access-2qptv\") pod \"payload-pre-processing-7b98bb678c-74wfk\" (UID: \"e8d20065-2c84-4fbf-8cca-39aaf998c068\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.165453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.165423 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:24.171680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.171657 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:24.171764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.171706 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:24.188221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.188192 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.188546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.188246 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9vs\" (UniqueName: \"kubernetes.io/projected/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-kube-api-access-vc9vs\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.188864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.188845 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.257581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.257517 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9vs\" (UniqueName: \"kubernetes.io/projected/02cf8231-7c60-4b6f-bc1d-07dbfc48a57a-kube-api-access-vc9vs\") pod \"payload-processing-55fbbbf9dc-j4tg5\" (UID: \"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.282189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.282158 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:24.548842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.533709 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-74wfk"] Nov 02 15:06:24.552815 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:06:24.552776 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d20065_2c84_4fbf_8cca_39aaf998c068.slice/crio-e9312e66f3282986cf1eedfb51b7bb97123f57a706690bf835858c7edf55897e WatchSource:0}: Error finding container e9312e66f3282986cf1eedfb51b7bb97123f57a706690bf835858c7edf55897e: Status 404 returned error can't find the container with id e9312e66f3282986cf1eedfb51b7bb97123f57a706690bf835858c7edf55897e Nov 02 15:06:24.675237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.675201 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j4tg5"] Nov 02 15:06:24.677581 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:06:24.677557 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cf8231_7c60_4b6f_bc1d_07dbfc48a57a.slice/crio-b7188802accedbd91c0c910aee379e9288d120cd04753dc1bc4284ddf313fcc6 WatchSource:0}: Error finding container b7188802accedbd91c0c910aee379e9288d120cd04753dc1bc4284ddf313fcc6: Status 404 returned error can't find the container with id b7188802accedbd91c0c910aee379e9288d120cd04753dc1bc4284ddf313fcc6 Nov 02 15:06:24.769042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.769003 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" event={"ID":"e8d20065-2c84-4fbf-8cca-39aaf998c068","Type":"ContainerStarted","Data":"e9312e66f3282986cf1eedfb51b7bb97123f57a706690bf835858c7edf55897e"} Nov 02 15:06:24.770085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:24.770065 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" event={"ID":"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a","Type":"ContainerStarted","Data":"b7188802accedbd91c0c910aee379e9288d120cd04753dc1bc4284ddf313fcc6"} Nov 02 15:06:25.172168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:25.172135 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:25.172316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:25.172187 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:26.052574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:26.052532 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:26.172825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:26.172384 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:26.172825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:26.172443 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:27.172396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.171979 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 15:06:27.172396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.172037 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 15:06:27.789866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.789824 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" event={"ID":"e8d20065-2c84-4fbf-8cca-39aaf998c068","Type":"ContainerStarted","Data":"7721717890c4f4f2fdec3936bc9c16950e39b5ddf1b90bba7865cd2266665c23"} Nov 02 15:06:27.790079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.789894 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:27.791510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.791481 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" event={"ID":"02cf8231-7c60-4b6f-bc1d-07dbfc48a57a","Type":"ContainerStarted","Data":"d5afdb6f0bb6dd62e270bb3972383c35afe1cc876bb9081054e5627c790f5991"} Nov 02 15:06:27.791654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:27.791639 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:28.021937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:28.021876 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" podStartSLOduration=2.163006238 podStartE2EDuration="5.021854822s" totalImagesPullingTime="2.858848584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:28.021414031 +0000 UTC m=+590.905016553" watchObservedRunningTime="2026-11-02 15:06:28.021854822 +0000 UTC m=+590.905457343" Nov 02 15:06:28.023592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:28.023537 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" podStartSLOduration=2.212480949 podStartE2EDuration="5.023523901s" totalImagesPullingTime="2.811042952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:27.905847792 +0000 UTC m=+590.789450312" watchObservedRunningTime="2026-11-02 15:06:28.023523901 +0000 UTC m=+590.907126417" Nov 02 15:06:28.369388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:28.369354 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:28.369869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:28.369762 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:28.370790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:28.370770 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:06:35.324655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:35.324594 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:36.184680 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:06:36.184642 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:06:38.800036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:38.800009 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-74wfk" Nov 02 15:06:38.800495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:38.800073 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-j4tg5" Nov 02 15:06:46.266203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:46.266157 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:46.707952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:46.707873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:48.359963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:48.359925 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:48.580737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:48.580704 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:50.095949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:50.095910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:50.307176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:50.307146 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:52.207063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:06:52.207031 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:06.185374 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:06.185337 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:07:14.662562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:14.662523 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:14.868525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:14.868489 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:15.176306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.176273 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:07:15.183060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.183031 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.249121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.249093 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de35cd-openshift-default-dockercfg-wdrm7\"" Nov 02 15:07:15.275534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275505 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275539 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275556 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ksx\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275682 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275719 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275745 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275871 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275900 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.275961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.275926 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.298155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.298130 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:07:15.353329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.353302 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:15.376546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376516 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376556 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376582 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376623 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376650 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376676 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376742 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376766 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.376888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.376792 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ksx\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.377088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.377044 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.377146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.377074 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.377146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.377124 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.377316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.377294 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.377431 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.377341 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.379177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.379152 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.379301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.379284 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.454139 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.454058 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ksx\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.466416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.466388 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.497044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.497015 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:15.690984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.690947 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:15.925713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.925689 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:07:15.927920 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:15.927892 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e95c7e_2901_4a66_aa55_66c0587cab08.slice/crio-37d059c703106a347066889f48b82ff39ce374434198d7ea53acd6da1abe7313 WatchSource:0}: Error finding container 37d059c703106a347066889f48b82ff39ce374434198d7ea53acd6da1abe7313: Status 404 returned error can't find the container with id 37d059c703106a347066889f48b82ff39ce374434198d7ea53acd6da1abe7313 Nov 02 15:07:15.930409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.930368 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:15.930492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.930444 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:15.930492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.930473 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:15.936328 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.936307 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:07:15.941331 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.941278 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.949615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.949591 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983185 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983248 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983293 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfllc\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983343 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983406 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983447 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983476 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983503 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.983880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.983543 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:15.993536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:15.993455 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ba49d0-openshift-default-dockercfg-66tsc\"" Nov 02 15:07:16.003056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.003033 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:07:16.015838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.010053 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerStarted","Data":"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f"} Nov 02 15:07:16.015838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.010113 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerStarted","Data":"37d059c703106a347066889f48b82ff39ce374434198d7ea53acd6da1abe7313"} Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084222 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084289 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084317 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084357 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084393 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.084601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084449 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.085091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.084870 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.085091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.085043 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.085204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.085091 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfllc\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.085204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.085175 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.086054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.086026 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.086272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.086252 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.087196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.087172 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.087775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.087742 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.091204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.091181 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.093626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.093601 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.146575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.146535 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.148193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.148163 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfllc\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.163318 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.163278 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:16.185051 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.184968 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podStartSLOduration=1.184955462 podStartE2EDuration="1.184955462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:16.1825941 +0000 UTC m=+639.066196620" watchObservedRunningTime="2026-11-02 15:07:16.184955462 +0000 UTC m=+639.068557972" Nov 02 15:07:16.267765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.267732 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:16.396274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.396229 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:16.497843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.497813 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:16.502762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.502737 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:16.588686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.588656 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:07:16.593983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.593963 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.652551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.652524 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6dc69b-openshift-default-dockercfg-pbd2c\"" Nov 02 15:07:16.653474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.653450 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:16.666218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.666196 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:07:16.690560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690532 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690603 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690628 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690664 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690742 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrcg\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690778 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690836 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.690874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690865 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.691391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.690888 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.702847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.702828 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:07:16.706101 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:16.706071 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0222cad_49d2_43f0_afad_e48e5d589046.slice/crio-f77723b27a7d088576f476128ea0f9783c912d4ac3cf94a45196c77329bffabf WatchSource:0}: Error finding container f77723b27a7d088576f476128ea0f9783c912d4ac3cf94a45196c77329bffabf: Status 404 returned error can't find the container with id f77723b27a7d088576f476128ea0f9783c912d4ac3cf94a45196c77329bffabf Nov 02 15:07:16.708417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.708381 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:16.708481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.708466 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:16.708527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.708512 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:16.792009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.791980 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792015 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792050 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792074 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrcg\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792101 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792130 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792391 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792411 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792451 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792505 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792522 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792873 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.792986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.792966 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.793158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.793140 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.794657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.794634 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.794881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.794863 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.853225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.853196 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.853357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.853279 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrcg\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:16.859664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.859633 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:16.906772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:16.906741 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:17.021863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.021662 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerStarted","Data":"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d"} Nov 02 15:07:17.021863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.021724 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerStarted","Data":"f77723b27a7d088576f476128ea0f9783c912d4ac3cf94a45196c77329bffabf"} Nov 02 15:07:17.021863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.021758 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:17.021863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.021832 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:07:17.084751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.084719 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:17.147852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.147771 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podStartSLOduration=2.147752078 podStartE2EDuration="2.147752078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:17.146034355 +0000 UTC m=+640.029636875" watchObservedRunningTime="2026-11-02 15:07:17.147752078 +0000 UTC m=+640.031354598" Nov 02 15:07:17.268890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.268857 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:17.274383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.274317 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:17.372092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.372041 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:07:17.374577 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:17.374549 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba1d179_ef29_433d_9a5f_5916d7afd14a.slice/crio-52b46b619b65c08ee5068d024e1b00b1e064abe715a13eb9327b4ddc83effe37 WatchSource:0}: Error finding container 52b46b619b65c08ee5068d024e1b00b1e064abe715a13eb9327b4ddc83effe37: Status 404 returned error can't find the container with id 52b46b619b65c08ee5068d024e1b00b1e064abe715a13eb9327b4ddc83effe37 Nov 02 15:07:17.375950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.375922 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:17.377321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.377289 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:17.377430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.377362 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:17.377430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.377408 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:17.614238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.614156 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:17.838831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:17.838767 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:18.026093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.026055 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerStarted","Data":"ca15cdbf6e3c5ab0fddf0ac7da86c7709e89fbb7d5b3799cdd8e9da2e75596cf"} Nov 02 15:07:18.026285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.026149 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerStarted","Data":"52b46b619b65c08ee5068d024e1b00b1e064abe715a13eb9327b4ddc83effe37"} Nov 02 15:07:18.026612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.026594 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:18.027422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.027403 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:18.150491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.150439 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podStartSLOduration=2.150425047 podStartE2EDuration="2.150425047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:18.148631023 +0000 UTC m=+641.032233545" watchObservedRunningTime="2026-11-02 15:07:18.150425047 +0000 UTC m=+641.034027565" Nov 02 15:07:18.907436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.907404 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:18.913135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:18.913111 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:19.030578 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:19.030546 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:19.031621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:19.031597 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:07:19.552536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:19.552497 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:19.754746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:19.754709 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:20.188671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:20.188640 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:20.419145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:20.419106 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:20.987930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:20.987901 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:21.193696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:21.193662 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:21.766534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:21.766494 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:21.971593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:21.971553 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.056867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.056823 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.259247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.259218 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.462617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.462587 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.508673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.508644 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:07:24.512262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.512243 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.572481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.572441 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxhz\" (UniqueName: \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.572627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.572602 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.577272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.577235 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:07:24.624660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.624609 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-de35cd\"" Nov 02 15:07:24.624660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.624643 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-de35cd-dockercfg-m5tsf\"" Nov 02 15:07:24.673628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.673586 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.673845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.673680 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxhz\" (UniqueName: \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.674767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.674743 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.675673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.675654 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:07:24.679260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.679246 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.701329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.701308 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.734474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.734417 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxhz\" (UniqueName: \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\") pod \"payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.740954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.740926 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:07:24.774720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.774700 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlrl\" (UniqueName: \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.774874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.774740 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.825039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.825016 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:24.876243 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.876210 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlrl\" (UniqueName: \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.876316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.876289 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.877833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.877281 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.916559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.916522 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:24.938364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.938336 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlrl\" (UniqueName: \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\") pod \"payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:24.994047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:24.993977 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:25.194967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:25.194944 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:07:25.200484 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:25.200455 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df298a8_465a_41e9_9902_c01eccae7da5.slice/crio-da3af2477e56c42bc5a9cdc9aa9356446022bf1984ef468685716d746c92fb37 WatchSource:0}: Error finding container da3af2477e56c42bc5a9cdc9aa9356446022bf1984ef468685716d746c92fb37: Status 404 returned error can't find the container with id da3af2477e56c42bc5a9cdc9aa9356446022bf1984ef468685716d746c92fb37 Nov 02 15:07:25.382821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:25.382776 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:07:25.386438 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:25.386414 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a0590_726f_488c_9af7_7c4d55d554fb.slice/crio-2d109047fe7d562ed47ff84bebd935a73f4a1cb0bb7c5fe41244d77bcecd09f8 WatchSource:0}: Error finding container 2d109047fe7d562ed47ff84bebd935a73f4a1cb0bb7c5fe41244d77bcecd09f8: Status 404 returned error can't find the container with id 2d109047fe7d562ed47ff84bebd935a73f4a1cb0bb7c5fe41244d77bcecd09f8 Nov 02 15:07:26.003992 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.003963 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:26.077385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.077347 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" event={"ID":"929a0590-726f-488c-9af7-7c4d55d554fb","Type":"ContainerStarted","Data":"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366"} Nov 02 15:07:26.077571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.077412 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" event={"ID":"929a0590-726f-488c-9af7-7c4d55d554fb","Type":"ContainerStarted","Data":"2d109047fe7d562ed47ff84bebd935a73f4a1cb0bb7c5fe41244d77bcecd09f8"} Nov 02 15:07:26.077571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.077547 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:26.078833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.078813 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" event={"ID":"7df298a8-465a-41e9-9902-c01eccae7da5","Type":"ContainerStarted","Data":"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9"} Nov 02 15:07:26.078980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.078848 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" event={"ID":"7df298a8-465a-41e9-9902-c01eccae7da5","Type":"ContainerStarted","Data":"da3af2477e56c42bc5a9cdc9aa9356446022bf1984ef468685716d746c92fb37"} Nov 02 15:07:26.078980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.078935 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:26.194511 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.194463 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" podStartSLOduration=2.194449574 podStartE2EDuration="2.194449574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:26.19173085 +0000 UTC m=+649.075333370" watchObservedRunningTime="2026-11-02 15:07:26.194449574 +0000 UTC m=+649.078052142" Nov 02 15:07:26.209653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.209623 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:26.308241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.308145 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" podStartSLOduration=2.308128324 podStartE2EDuration="2.308128324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:26.306986895 +0000 UTC m=+649.190589414" watchObservedRunningTime="2026-11-02 15:07:26.308128324 +0000 UTC m=+649.191730844" Nov 02 15:07:26.411750 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.411718 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:26.624437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:26.624359 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:28.763081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:28.763046 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.006202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.006171 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.246615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.246582 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.435353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.435321 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:07:29.439058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.439041 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.461095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.461068 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.491005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.490982 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-db3931\"" Nov 02 15:07:29.524705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.524623 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.524872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.524746 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrqt\" (UniqueName: \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.541109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.541034 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-db3931-dockercfg-wpr4q\"" Nov 02 15:07:29.546869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.546841 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:07:29.591949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.591917 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:07:29.595495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.595477 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.626078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.626050 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.626196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.626098 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrqt\" (UniqueName: \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.626716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.626693 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.633217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.633190 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:07:29.687230 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.687196 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrqt\" (UniqueName: \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\") pod \"payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.726699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.726672 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.726900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.726880 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tfw\" (UniqueName: \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.752540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.752519 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:29.828616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.828573 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tfw\" (UniqueName: \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.829054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.828747 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.829634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.829609 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.847650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.847618 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.890064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.890034 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tfw\" (UniqueName: \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\") pod \"payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:29.908465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:29.908445 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:30.128006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:30.127983 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:07:30.129286 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:30.129260 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd1784a_b0ba_490d_83c5_3171a145cbd5.slice/crio-427ce8d20e6af38135573ffdb4bf73c445e92c7f2d4d0940e6f2f6c2d8dd04db WatchSource:0}: Error finding container 427ce8d20e6af38135573ffdb4bf73c445e92c7f2d4d0940e6f2f6c2d8dd04db: Status 404 returned error can't find the container with id 427ce8d20e6af38135573ffdb4bf73c445e92c7f2d4d0940e6f2f6c2d8dd04db Nov 02 15:07:30.279473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:30.279450 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:07:30.281467 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:30.281445 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225d6774_62eb_46c1_b360_9a6af913b885.slice/crio-150972d7e471d24be010b1d6b65976fdf3178225cd5eea612acd94f9f3658fbf WatchSource:0}: Error finding container 150972d7e471d24be010b1d6b65976fdf3178225cd5eea612acd94f9f3658fbf: Status 404 returned error can't find the container with id 150972d7e471d24be010b1d6b65976fdf3178225cd5eea612acd94f9f3658fbf Nov 02 15:07:31.062069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.062037 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:07:31.065765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.065746 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.102507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.102479 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" event={"ID":"225d6774-62eb-46c1-b360-9a6af913b885","Type":"ContainerStarted","Data":"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1"} Nov 02 15:07:31.102642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.102534 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" event={"ID":"225d6774-62eb-46c1-b360-9a6af913b885","Type":"ContainerStarted","Data":"150972d7e471d24be010b1d6b65976fdf3178225cd5eea612acd94f9f3658fbf"} Nov 02 15:07:31.102642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.102566 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:31.104088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.104060 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" event={"ID":"9fd1784a-b0ba-490d-83c5-3171a145cbd5","Type":"ContainerStarted","Data":"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3"} Nov 02 15:07:31.104174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.104099 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" event={"ID":"9fd1784a-b0ba-490d-83c5-3171a145cbd5","Type":"ContainerStarted","Data":"427ce8d20e6af38135573ffdb4bf73c445e92c7f2d4d0940e6f2f6c2d8dd04db"} Nov 02 15:07:31.104174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.104155 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:31.140421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.140383 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f2n\" (UniqueName: \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.140586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.140568 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.140709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.140685 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.140751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.140735 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.168542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.168511 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ba49d0-serving-cert\"" Nov 02 15:07:31.168711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.168602 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:07:31.168711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.168602 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:07:31.168711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.168633 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vkjdq\"" Nov 02 15:07:31.168711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.168704 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:07:31.171199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.171175 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:07:31.241782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.241753 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.241949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.241822 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.241949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.241860 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.242083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.241942 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f2n\" (UniqueName: \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.242083 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:31.241999 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-serving-cert: secret "maas-api-e2e-worker-w0-ba49d0-serving-cert" not found Nov 02 15:07:31.242083 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:31.242070 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls podName:4ec70061-42bd-4543-b001-3c4298b5ae78 nodeName:}" failed. No retries permitted until 2026-11-02 15:07:31.742050317 +0000 UTC m=+654.625652835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls") pod "maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" (UID: "4ec70061-42bd-4543-b001-3c4298b5ae78") : secret "maas-api-e2e-worker-w0-ba49d0-serving-cert" not found Nov 02 15:07:31.242605 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.242578 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.244424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.244403 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.304150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.304071 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f2n\" (UniqueName: \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.315672 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.315584 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:31.333663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.333608 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" podStartSLOduration=2.333588501 podStartE2EDuration="2.333588501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:31.332599669 +0000 UTC m=+654.216202190" watchObservedRunningTime="2026-11-02 15:07:31.333588501 +0000 UTC m=+654.217191023" Nov 02 15:07:31.449301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.449246 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" podStartSLOduration=2.449231785 podStartE2EDuration="2.449231785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:31.447064091 +0000 UTC m=+654.330666613" watchObservedRunningTime="2026-11-02 15:07:31.449231785 +0000 UTC m=+654.332834304" Nov 02 15:07:31.522003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.521970 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:31.735512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.735478 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:31.747559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.747531 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.749856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.749834 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:31.940954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.940912 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:31.967683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.964844 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:07:31.970494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.970472 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:31.979395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:31.979378 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:32.021199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.021164 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:07:32.033845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.033301 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ba49d0\"" Nov 02 15:07:32.038251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.038227 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ba49d0-dockercfg-2nrvc\"" Nov 02 15:07:32.059322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.055336 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9ct\" (UniqueName: \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.059322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.055447 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.152152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.152118 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:07:32.156719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.156699 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.156868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.156846 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9ct\" (UniqueName: \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.156933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.156923 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.157556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.157539 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.216450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.216425 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:07:32.223715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.223687 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9ct\" (UniqueName: \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\") pod \"payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.257916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.257856 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.258036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.258018 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzckw\" (UniqueName: \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.283278 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.283252 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:32.359471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.359436 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzckw\" (UniqueName: \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.359621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.359598 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.360512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.360485 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.403956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.403932 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:07:32.405682 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:32.405659 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec70061_42bd_4543_b001_3c4298b5ae78.slice/crio-262d339bdbc145af59540667f59d88faf87c42ab380690b8e6354b1e5408933b WatchSource:0}: Error finding container 262d339bdbc145af59540667f59d88faf87c42ab380690b8e6354b1e5408933b: Status 404 returned error can't find the container with id 262d339bdbc145af59540667f59d88faf87c42ab380690b8e6354b1e5408933b Nov 02 15:07:32.407070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.407052 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:07:32.420347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.420329 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzckw\" (UniqueName: \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\") pod \"payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.470523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.470498 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:32.654381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.654359 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:07:32.656361 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:32.656329 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf67fec_d305_44f3_ad86_73424427b88e.slice/crio-355874bb971ba4769491113d8cddf7d14d122a4a10444602dfdd824128f5c2f8 WatchSource:0}: Error finding container 355874bb971ba4769491113d8cddf7d14d122a4a10444602dfdd824128f5c2f8: Status 404 returned error can't find the container with id 355874bb971ba4769491113d8cddf7d14d122a4a10444602dfdd824128f5c2f8 Nov 02 15:07:32.840944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:32.840896 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:07:32.843565 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:32.843536 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44660d8c_1252_4503_8303_afffee9fb66d.slice/crio-74689d21a181e53c0e61c7b3764f5bc715e610e001a6a813188ed498b0c3d64e WatchSource:0}: Error finding container 74689d21a181e53c0e61c7b3764f5bc715e610e001a6a813188ed498b0c3d64e: Status 404 returned error can't find the container with id 74689d21a181e53c0e61c7b3764f5bc715e610e001a6a813188ed498b0c3d64e Nov 02 15:07:33.116461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.116362 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" event={"ID":"6bf67fec-d305-44f3-ad86-73424427b88e","Type":"ContainerStarted","Data":"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4"} Nov 02 15:07:33.116461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.116427 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" event={"ID":"6bf67fec-d305-44f3-ad86-73424427b88e","Type":"ContainerStarted","Data":"355874bb971ba4769491113d8cddf7d14d122a4a10444602dfdd824128f5c2f8"} Nov 02 15:07:33.116637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.116547 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:33.118061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.118037 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" event={"ID":"44660d8c-1252-4503-8303-afffee9fb66d","Type":"ContainerStarted","Data":"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b"} Nov 02 15:07:33.118160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.118077 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" event={"ID":"44660d8c-1252-4503-8303-afffee9fb66d","Type":"ContainerStarted","Data":"74689d21a181e53c0e61c7b3764f5bc715e610e001a6a813188ed498b0c3d64e"} Nov 02 15:07:33.118160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.118114 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:33.119280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.119253 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" event={"ID":"4ec70061-42bd-4543-b001-3c4298b5ae78","Type":"ContainerStarted","Data":"262d339bdbc145af59540667f59d88faf87c42ab380690b8e6354b1e5408933b"} Nov 02 15:07:33.235557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.235496 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" podStartSLOduration=2.2354791 podStartE2EDuration="2.2354791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:33.232048465 +0000 UTC m=+656.115651009" watchObservedRunningTime="2026-11-02 15:07:33.2354791 +0000 UTC m=+656.119081619" Nov 02 15:07:33.347849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:33.347773 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" podStartSLOduration=1.3477544080000001 podStartE2EDuration="1.347754408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:33.346826916 +0000 UTC m=+656.230429631" watchObservedRunningTime="2026-11-02 15:07:33.347754408 +0000 UTC m=+656.231356928" Nov 02 15:07:34.140150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.140074 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" event={"ID":"4ec70061-42bd-4543-b001-3c4298b5ae78","Type":"ContainerStarted","Data":"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac"} Nov 02 15:07:34.140719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.140697 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:34.242442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.242408 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:34.259863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.259816 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" podStartSLOduration=2.402046861 podStartE2EDuration="3.259788874s" totalImagesPullingTime="857.742013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:34.258515078 +0000 UTC m=+657.142117598" watchObservedRunningTime="2026-11-02 15:07:34.259788874 +0000 UTC m=+657.143391392" Nov 02 15:07:34.384161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.384128 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:07:34.448315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:34.448235 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:35.670703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:35.670668 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:36.185124 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:36.185085 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:07:36.602916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:36.602885 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:36.805848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:36.805793 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:37.009381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:37.009351 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:37.085998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:37.085973 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:07:37.086139 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:37.086071 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:07:39.054083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.054045 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:39.266168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.266130 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:39.411704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.411627 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:07:39.416540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.416521 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.468669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.468644 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-42555d\"" Nov 02 15:07:39.468775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.468669 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-42555d-dockercfg-mqchj\"" Nov 02 15:07:39.476062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.476037 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:07:39.548280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.548247 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.548460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.548438 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdz9\" (UniqueName: \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.568234 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.568199 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:07:39.572113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.572093 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.632078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.632046 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:07:39.649849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.649815 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdz9\" (UniqueName: \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.650007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.649914 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.650690 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.650665 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.710343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.710318 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdz9\" (UniqueName: \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\") pod \"payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.730156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.730132 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:39.751647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.751613 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsm69\" (UniqueName: \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.751903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.751878 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.853501 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.853463 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.853661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.853630 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsm69\" (UniqueName: \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.854225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.854205 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:39.926396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:39.926357 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsm69\" (UniqueName: \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\") pod \"payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:40.104867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.104845 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:07:40.107602 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:40.107576 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0528b4_a52b_4f8c_8613_144e25e9c95f.slice/crio-3213a78a0388a9c41b3fc60eea08f8398dc7443e2952c333678cd4a6f3bcba99 WatchSource:0}: Error finding container 3213a78a0388a9c41b3fc60eea08f8398dc7443e2952c333678cd4a6f3bcba99: Status 404 returned error can't find the container with id 3213a78a0388a9c41b3fc60eea08f8398dc7443e2952c333678cd4a6f3bcba99 Nov 02 15:07:40.179320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.179290 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" event={"ID":"de0528b4-a52b-4f8c-8613-144e25e9c95f","Type":"ContainerStarted","Data":"ec01e8a7d3bbc80e82dff2e2f3b051570beedb6504329c03910ea538a7f4c5a7"} Nov 02 15:07:40.179435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.179356 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:40.179435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.179384 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" event={"ID":"de0528b4-a52b-4f8c-8613-144e25e9c95f","Type":"ContainerStarted","Data":"3213a78a0388a9c41b3fc60eea08f8398dc7443e2952c333678cd4a6f3bcba99"} Nov 02 15:07:40.196783 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.196761 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:40.298252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.298199 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" podStartSLOduration=1.298183778 podStartE2EDuration="1.298183778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:40.295006579 +0000 UTC m=+663.178609099" watchObservedRunningTime="2026-11-02 15:07:40.298183778 +0000 UTC m=+663.181786297" Nov 02 15:07:40.592652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:40.592621 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:07:40.594380 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:40.594356 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4c08af_9592_414b_945e_6497bc057e57.slice/crio-1c30efd4dd9b0fee30555bd0a95d4ee746b06fecdfaffa19a62c829b0f5ceb42 WatchSource:0}: Error finding container 1c30efd4dd9b0fee30555bd0a95d4ee746b06fecdfaffa19a62c829b0f5ceb42: Status 404 returned error can't find the container with id 1c30efd4dd9b0fee30555bd0a95d4ee746b06fecdfaffa19a62c829b0f5ceb42 Nov 02 15:07:41.157778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.157750 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:07:41.185443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.185411 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" event={"ID":"ab4c08af-9592-414b-945e-6497bc057e57","Type":"ContainerStarted","Data":"4e368d5db8e9a7d49f5a12dd565b74cb6b2899359d2b39c5e3d4919357456394"} Nov 02 15:07:41.185585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.185461 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" event={"ID":"ab4c08af-9592-414b-945e-6497bc057e57","Type":"ContainerStarted","Data":"1c30efd4dd9b0fee30555bd0a95d4ee746b06fecdfaffa19a62c829b0f5ceb42"} Nov 02 15:07:41.391505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.391452 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" podStartSLOduration=2.3914383900000002 podStartE2EDuration="2.39143839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:41.390252148 +0000 UTC m=+664.273854667" watchObservedRunningTime="2026-11-02 15:07:41.39143839 +0000 UTC m=+664.275040909" Nov 02 15:07:41.584330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.584294 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:41.785744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.785694 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:41.993225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:41.993191 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:42.113927 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:42.113791 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:07:42.114061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:42.113988 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:07:42.195202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:42.195171 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:43.386457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:43.386420 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:07:44.141951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:44.141921 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:07:44.143269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:44.143246 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:07:46.511150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.511121 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:46.719425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.719387 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:46.874524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.874447 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:07:46.879372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.879353 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:46.942152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.942125 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6dc69b\"" Nov 02 15:07:46.966077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.966040 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:46.966337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.966315 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr72c\" (UniqueName: \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:46.994044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.994014 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6dc69b-dockercfg-p7ppf\"" Nov 02 15:07:46.996847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:46.996823 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:07:47.067370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.067338 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:47.067532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.067387 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr72c\" (UniqueName: \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:47.068052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.068032 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:47.152498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.152423 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr72c\" (UniqueName: \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\") pod \"payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:47.193993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.193966 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:47.427885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.424218 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:47.612361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.612279 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:07:47.653826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.652451 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:47.855993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:47.855913 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:48.251659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:48.251615 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" event={"ID":"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9","Type":"ContainerStarted","Data":"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d"} Nov 02 15:07:48.251890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:48.251681 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" event={"ID":"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9","Type":"ContainerStarted","Data":"06715cfd7a7af83691bf8f3e09b71e366e8e4c182caec5adf817f072401adb11"} Nov 02 15:07:48.251890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:48.251725 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:48.367861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:48.367790 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" podStartSLOduration=2.367773704 podStartE2EDuration="2.367773704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:48.366214393 +0000 UTC m=+671.249816916" watchObservedRunningTime="2026-11-02 15:07:48.367773704 +0000 UTC m=+671.251376223" Nov 02 15:07:51.189116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:51.189081 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:07:52.571154 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.571119 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:07:52.576926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.576905 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.630635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.630601 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-019efe-serving-cert\"" Nov 02 15:07:52.630778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.630705 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.630778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.630742 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.630778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.630767 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.630920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.630786 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przwh\" (UniqueName: \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.642701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.642672 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:07:52.732243 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.732192 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.732439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.732270 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.732439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.732312 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.732439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.732341 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przwh\" (UniqueName: \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.732606 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:52.732450 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-serving-cert: secret "maas-api-e2e-worker-w1-019efe-serving-cert" not found Nov 02 15:07:52.732606 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:07:52.732546 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls podName:17eced10-4d6a-4f91-ae37-ebcfec5e8ac5 nodeName:}" failed. No retries permitted until 2026-11-02 15:07:53.23252112 +0000 UTC m=+676.116123636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls") pod "maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" (UID: "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5") : secret "maas-api-e2e-worker-w1-019efe-serving-cert" not found Nov 02 15:07:52.733052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.733024 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.734900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.734875 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.794255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.794220 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przwh\" (UniqueName: \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:52.895114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:52.894953 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:53.102837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.102738 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:53.204466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.204382 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:07:53.236854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.236820 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:53.239328 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.239306 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:53.278046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.278020 4469 generic.go:356] "Generic (PLEG): container finished" podID="d0222cad-49d2-43f0-afad-e48e5d589046" containerID="71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d" exitCode=0 Nov 02 15:07:53.278179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.278089 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerDied","Data":"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d"} Nov 02 15:07:53.278459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.278445 4469 scope.go:122] "RemoveContainer" containerID="71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d" Nov 02 15:07:53.279723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.279697 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:53.279817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.279759 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:53.279817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.279790 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:07:53.304711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.304678 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:53.492027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.492000 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:53.875760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:53.875728 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:07:53.877553 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:07:53.877526 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eced10_4d6a_4f91_ae37_ebcfec5e8ac5.slice/crio-e738b7bbdfc7ec24bfeb6400780a2a5b27f0724b0d4c641382a523f985f46256 WatchSource:0}: Error finding container e738b7bbdfc7ec24bfeb6400780a2a5b27f0724b0d4c641382a523f985f46256: Status 404 returned error can't find the container with id e738b7bbdfc7ec24bfeb6400780a2a5b27f0724b0d4c641382a523f985f46256 Nov 02 15:07:54.285585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:54.285545 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerStarted","Data":"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302"} Nov 02 15:07:54.286790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:54.286759 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" event={"ID":"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5","Type":"ContainerStarted","Data":"e738b7bbdfc7ec24bfeb6400780a2a5b27f0724b0d4c641382a523f985f46256"} Nov 02 15:07:54.354822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:54.353892 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:07:55.268432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:55.268407 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:07:55.268873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:55.268853 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:55.268920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:55.268901 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 15:07:55.293796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:55.293764 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" event={"ID":"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5","Type":"ContainerStarted","Data":"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a"} Nov 02 15:07:55.293985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:55.293972 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:07:56.268836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:56.268262 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:56.268836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:56.268345 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 15:07:57.275201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:57.274255 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:57.275201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:57.274320 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 15:07:58.274840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:58.274523 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:58.274840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:58.274578 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 15:07:58.395434 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:58.395377 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:59.278826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:59.277888 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:59.278826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:59.277962 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 15:07:59.284827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:59.280678 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:07:59.412152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:07:59.412081 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" podStartSLOduration=6.526443967 podStartE2EDuration="7.412065102s" totalImagesPullingTime="885.621135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:55.462136676 +0000 UTC m=+678.345739195" watchObservedRunningTime="2026-11-02 15:07:59.412065102 +0000 UTC m=+682.295667683" Nov 02 15:08:00.272553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:00.272523 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:08:00.272940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:00.272918 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:08:00.273692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:00.273674 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:08:01.305269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:01.305238 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:08:03.067845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:03.060960 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:08:03.067845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:03.061032 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:08:03.278039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:03.278004 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:08:03.376472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:03.376382 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:04.341684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.341653 4469 generic.go:356] "Generic (PLEG): container finished" podID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerID="854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f" exitCode=0 Nov 02 15:08:04.342127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.341727 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerDied","Data":"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f"} Nov 02 15:08:04.342127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.342086 4469 scope.go:122] "RemoveContainer" containerID="854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f" Nov 02 15:08:04.343388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.343363 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:04.343449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.343428 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:04.343482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.343459 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:04.978424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:04.978392 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:05.356138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.356052 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerStarted","Data":"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62"} Nov 02 15:08:05.497445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.497413 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:08:05.497633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.497611 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 15:08:05.497686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.497654 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 15:08:05.587815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.587770 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:05.800313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:05.800273 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:06.186089 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:08:06.186006 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:08:06.497544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:06.497507 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 15:08:06.497960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:06.497563 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 15:08:07.497330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:07.497292 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 15:08:07.497540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:07.497341 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 15:08:08.498353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:08.498318 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 15:08:08.498743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:08.498393 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 15:08:08.689862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:08.689785 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:08.896212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:08.896123 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:09.147885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:09.147772 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:09.497366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:09.497333 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 15:08:09.497537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:09.497391 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 15:08:10.501385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:10.501353 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:08:10.501867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:10.501597 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:08:10.502246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:10.502228 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:08:10.892035 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:10.891930 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:11.554128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:11.554089 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:11.830031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:11.829957 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:13.745845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.745812 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:08:13.749127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.749105 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.800912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.800879 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-c18e79-serving-cert\"" Nov 02 15:08:13.817793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.817766 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.817935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.817834 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.817981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.817963 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.818034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.818016 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzg2l\" (UniqueName: \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.852751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.852724 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:08:13.918765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.918726 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.918953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.918772 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzg2l\" (UniqueName: \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.918953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.918868 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.918953 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:08:13.918900 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-serving-cert: secret "maas-api-e2e-worker-w6-c18e79-serving-cert" not found Nov 02 15:08:13.918953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.918915 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.919195 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:08:13.918980 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls podName:88049ad1-b644-4eda-87cd-928cc3570c27 nodeName:}" failed. No retries permitted until 2026-11-02 15:08:14.418958717 +0000 UTC m=+697.302561228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls") pod "maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" (UID: "88049ad1-b644-4eda-87cd-928cc3570c27") : secret "maas-api-e2e-worker-w6-c18e79-serving-cert" not found Nov 02 15:08:13.919610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.919585 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.921540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.921515 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:13.980494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:13.980464 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzg2l\" (UniqueName: \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:14.008420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.008345 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:14.218587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.218557 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:14.424417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.424339 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:14.426289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.426253 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:14.426862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.426845 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:14.665317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:14.665283 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:15.051972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:15.051944 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:08:15.053374 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:08:15.053345 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88049ad1_b644_4eda_87cd_928cc3570c27.slice/crio-51ad7aa4abf9d82944070f2614aa9c56d4ce53201938544e2986a90377b86d10 WatchSource:0}: Error finding container 51ad7aa4abf9d82944070f2614aa9c56d4ce53201938544e2986a90377b86d10: Status 404 returned error can't find the container with id 51ad7aa4abf9d82944070f2614aa9c56d4ce53201938544e2986a90377b86d10 Nov 02 15:08:15.416035 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:15.414659 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" event={"ID":"88049ad1-b644-4eda-87cd-928cc3570c27","Type":"ContainerStarted","Data":"51ad7aa4abf9d82944070f2614aa9c56d4ce53201938544e2986a90377b86d10"} Nov 02 15:08:16.421772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:16.421721 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" event={"ID":"88049ad1-b644-4eda-87cd-928cc3570c27","Type":"ContainerStarted","Data":"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d"} Nov 02 15:08:16.422179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:16.421827 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:16.539666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:16.539607 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" podStartSLOduration=2.755322739 podStartE2EDuration="3.539590047s" totalImagesPullingTime="784.267308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:16.53905984 +0000 UTC m=+699.422662384" watchObservedRunningTime="2026-11-02 15:08:16.539590047 +0000 UTC m=+699.423192567" Nov 02 15:08:18.282094 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:18.282059 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:08:19.376553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:19.376517 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:22.452817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:22.452781 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:08:24.168995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:24.168959 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:26.518810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:26.518764 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:26.735529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:26.735490 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:27.121065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.121029 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:27.332825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.332739 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:27.503733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.503703 4469 generic.go:356] "Generic (PLEG): container finished" podID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerID="ca15cdbf6e3c5ab0fddf0ac7da86c7709e89fbb7d5b3799cdd8e9da2e75596cf" exitCode=0 Nov 02 15:08:27.503733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.503742 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerDied","Data":"ca15cdbf6e3c5ab0fddf0ac7da86c7709e89fbb7d5b3799cdd8e9da2e75596cf"} Nov 02 15:08:27.504136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.504111 4469 scope.go:122] "RemoveContainer" containerID="ca15cdbf6e3c5ab0fddf0ac7da86c7709e89fbb7d5b3799cdd8e9da2e75596cf" Nov 02 15:08:27.505399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.505364 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:27.505499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.505465 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:27.505546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:27.505511 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:08:28.511471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:28.511440 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerStarted","Data":"da04aa03d2f8266ca54c784ce9e8b3ddf742b232e77dc8c61b0dfa48cf93595b"} Nov 02 15:08:28.907471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:28.907382 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:08:28.907638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:28.907559 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:08:28.907716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:28.907627 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:08:29.763393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:29.763359 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:29.870108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:29.870072 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:08:29.907217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:29.907176 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:08:29.907393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:29.907240 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:08:30.907645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:30.907607 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:08:30.908062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:30.907661 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:08:31.907869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:31.907831 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:08:31.908271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:31.907889 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:08:32.908119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:32.908083 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:08:32.908495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:32.908137 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:08:33.912765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:33.912725 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:08:33.913172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:33.913089 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:08:33.913897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:33.913876 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:08:34.808081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:34.808041 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:35.697999 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:35.697963 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:35.904497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:35.904466 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:36.106792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:36.106761 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:36.187175 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:08:36.187143 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:08:36.323780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:36.323739 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:36.579794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:36.579751 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:36.959450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:36.959419 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:39.974834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:39.973737 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:41.897651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:41.897612 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:43.092262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:43.092224 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:43.686636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:43.686604 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:43.919959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:43.919921 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:45.404700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:45.404659 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:47.109577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:47.109535 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:50.355928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:50.355890 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:50.670401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:50.670315 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:08:55.150515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:55.150456 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:57.964964 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:57.964925 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:58.586555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:58.586517 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:58.798668 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:08:58.798627 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:00.357863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:00.357796 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:00.574570 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:00.574537 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:04.176047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:04.175996 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:04.792901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:04.792863 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:05.004546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:05.004511 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:05.678333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:05.678295 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:06.186549 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:09:06.186508 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:09:06.338911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:06.338881 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:09:10.651120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:10.651082 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:12.854378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:12.854341 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:15.451815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:15.451760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:16.856020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:16.855962 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:20.359937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:20.359839 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:25.149914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:25.149882 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:27.086470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:27.086423 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:28.862505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:28.862456 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:29.125211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:29.125130 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:09:29.964168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:29.964122 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:34.854756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:34.854715 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:36.185513 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:09:36.185480 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:09:36.367859 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:36.367815 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:37.301407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:37.301341 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:39.652864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:39.652824 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:42.179735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.179695 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:09:42.182677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.182654 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.284523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.284490 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"kube-root-ca.crt\"" Nov 02 15:09:42.284523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.284488 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"openshift-service-ca.crt\"" Nov 02 15:09:42.284727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.284491 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"default-dockercfg-ksn49\"" Nov 02 15:09:42.284727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.284490 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"premium-w0-ba49d0-kserve-self-signed-certs\"" Nov 02 15:09:42.287104 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.287086 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:09:42.288037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288017 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.288084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288069 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.288162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288142 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67cz\" (UniqueName: \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.288263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288250 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.288301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288283 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.288338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.288320 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.334855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.334791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:42.389828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.389782 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.389997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.389855 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.389997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.389897 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.389997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.389953 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.390167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.390008 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.390167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.390045 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67cz\" (UniqueName: \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.390328 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.390302 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.390399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.390331 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.390399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.390370 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.392146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.392125 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.392240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.392222 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.451659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.451595 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67cz\" (UniqueName: \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\") pod \"premium-w0-ba49d0-kserve-595d9c564-794d5\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.493741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.493717 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:42.865829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.865786 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:09:42.867896 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:09:42.867869 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2dbd84f_4471_454d_89d3_48f756e99e51.slice/crio-11297ad057b0482eea2101b04ca98b48ae21297a43f19d12cc240cccf927c2e1 WatchSource:0}: Error finding container 11297ad057b0482eea2101b04ca98b48ae21297a43f19d12cc240cccf927c2e1: Status 404 returned error can't find the container with id 11297ad057b0482eea2101b04ca98b48ae21297a43f19d12cc240cccf927c2e1 Nov 02 15:09:42.941834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.941786 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" event={"ID":"a2dbd84f-4471-454d-89d3-48f756e99e51","Type":"ContainerStarted","Data":"8b1989bca5cda958f4ba084deb2e140f6d49ae8936766f860f361fe2645b7180"} Nov 02 15:09:42.942018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:42.941852 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" event={"ID":"a2dbd84f-4471-454d-89d3-48f756e99e51","Type":"ContainerStarted","Data":"11297ad057b0482eea2101b04ca98b48ae21297a43f19d12cc240cccf927c2e1"} Nov 02 15:09:43.059170 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:43.059112 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podStartSLOduration=1.059098475 podStartE2EDuration="1.059098475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:43.058058541 +0000 UTC m=+785.941661061" watchObservedRunningTime="2026-11-02 15:09:43.059098475 +0000 UTC m=+785.942700993" Nov 02 15:09:43.493930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:43.493898 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:44.452566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:44.452529 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:45.375210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:45.375169 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:48.629274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:48.629235 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:09:49.397146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:49.397099 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:52.494884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:52.494851 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:52.507165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:52.507141 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:53.009913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:53.009880 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:09:54.251319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:54.251287 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:59.052299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:09:59.052265 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:02.234715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:02.234677 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:03.861354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:03.861312 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:06.185568 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:10:06.185534 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:10:08.758740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:08.758702 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:10.777105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:10.777070 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:11.159173 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:11.159078 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:10:13.543967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:13.543932 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:18.353600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:18.353567 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:23.157354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:23.157311 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:27.433827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:27.433779 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:28.013849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:28.013791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:32.851510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:32.851472 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:36.188188 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:10:36.188143 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:10:36.716664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:36.716632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:36.921297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:36.921262 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:37.645680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:37.645644 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:37.848038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:37.848001 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:42.539301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:42.539268 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:47.367370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:47.367327 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:48.643048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:48.643015 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:10:52.255116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:52.255079 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:57.049026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:10:57.048988 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:01.852895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:01.852859 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:02.456219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.456185 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:11:02.459925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.459901 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.513307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.513251 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db3931-openshift-default-dockercfg-w9t9c\"" Nov 02 15:11:02.532488 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532448 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532510 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532611 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4chw\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532639 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532676 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532743 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532865 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.532970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.532923 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.533059 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.533038 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.566298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.566270 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:11:02.633768 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.633734 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.633973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.633787 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.633973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.633904 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.633998 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4chw\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634037 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634078 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634102 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634143 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634201 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634363 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634580 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634688 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634635 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.634688 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.634666 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.635148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.635123 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.636635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.636611 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.636973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.636947 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.694909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.694882 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.695064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.695047 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4chw\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.725872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.725838 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:11:02.729468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.729450 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.734928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.734901 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.735055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.734974 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.735055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.735045 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.735179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.735089 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lvv\" (UniqueName: \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.735236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.735195 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.735236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.735220 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.773233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.773189 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:02.781876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.781782 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"openshift-service-ca.crt\"" Nov 02 15:11:02.782030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.781894 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"kube-root-ca.crt\"" Nov 02 15:11:02.782030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.781914 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"default-dockercfg-j445d\"" Nov 02 15:11:02.782137 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.782042 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"premium-w4-6dc69b-kserve-self-signed-certs\"" Nov 02 15:11:02.790359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.790328 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836055 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836119 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836175 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836258 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836293 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.836354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.836343 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lvv\" (UniqueName: \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.837465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.837438 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.837719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.837697 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.838454 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.837950 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.841653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.840580 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.841653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.841193 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:02.889498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.889458 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:02.898280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:02.898250 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lvv\" (UniqueName: \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\") pod \"premium-w4-6dc69b-kserve-5746d87898-48rtt\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:03.041440 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.041368 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:03.203257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.203234 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:11:03.204953 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:11:03.204927 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e42948_0742_419f_8e69_7836d124cdd0.slice/crio-0781ffa4ea0969aca9f5af02ea14881da2bb89c06b460db2d2b15467562ef50d WatchSource:0}: Error finding container 0781ffa4ea0969aca9f5af02ea14881da2bb89c06b460db2d2b15467562ef50d: Status 404 returned error can't find the container with id 0781ffa4ea0969aca9f5af02ea14881da2bb89c06b460db2d2b15467562ef50d Nov 02 15:11:03.207295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.207265 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:11:03.207366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.207326 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:11:03.207366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.207358 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:11:03.409302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.409215 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" event={"ID":"b9e42948-0742-419f-8e69-7836d124cdd0","Type":"ContainerStarted","Data":"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7"} Nov 02 15:11:03.409302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.409282 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" event={"ID":"b9e42948-0742-419f-8e69-7836d124cdd0","Type":"ContainerStarted","Data":"0781ffa4ea0969aca9f5af02ea14881da2bb89c06b460db2d2b15467562ef50d"} Nov 02 15:11:03.414347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.414326 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:11:03.415667 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:11:03.415641 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7697f54a_9b62_46cf_aa9d_aa753630d01b.slice/crio-b5710ed01094b46aee06ab2f7d42fc00a991c1763108785c3541f68948d5c13c WatchSource:0}: Error finding container b5710ed01094b46aee06ab2f7d42fc00a991c1763108785c3541f68948d5c13c: Status 404 returned error can't find the container with id b5710ed01094b46aee06ab2f7d42fc00a991c1763108785c3541f68948d5c13c Nov 02 15:11:03.535527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.535477 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podStartSLOduration=1.535463257 podStartE2EDuration="1.535463257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:03.533217629 +0000 UTC m=+866.416820149" watchObservedRunningTime="2026-11-02 15:11:03.535463257 +0000 UTC m=+866.419065776" Nov 02 15:11:03.773863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:03.773828 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:04.417005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:04.416962 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" event={"ID":"7697f54a-9b62-46cf-aa9d-aa753630d01b","Type":"ContainerStarted","Data":"80d5b8d1bddcb2dabee1ae205d04842bd794ad9b14506455554e09be2bc59da9"} Nov 02 15:11:04.417005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:04.417005 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" event={"ID":"7697f54a-9b62-46cf-aa9d-aa753630d01b","Type":"ContainerStarted","Data":"b5710ed01094b46aee06ab2f7d42fc00a991c1763108785c3541f68948d5c13c"} Nov 02 15:11:04.774810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:04.774774 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 15:11:04.774988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:04.774856 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 15:11:05.043881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:05.042882 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:05.773771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:05.773734 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 15:11:05.774282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:05.773793 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 15:11:06.186834 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:11:06.186736 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:11:06.774349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:06.774306 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 15:11:06.774758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:06.774381 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 15:11:06.863822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:06.859781 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:07.276691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:07.276655 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:07.479279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:07.479247 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:07.774452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:07.774414 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 15:11:07.774964 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:07.774484 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 15:11:08.773652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:08.773619 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 15:11:08.774824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:08.774773 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 15:11:09.884324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:09.884291 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:09.884814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:09.884598 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:09.885487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:09.885469 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:11:10.102822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:10.097591 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podStartSLOduration=8.097571948 podStartE2EDuration="8.097571948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:04.549473364 +0000 UTC m=+867.433075885" watchObservedRunningTime="2026-11-02 15:11:10.097571948 +0000 UTC m=+872.981174465" Nov 02 15:11:11.658267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:11.658218 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:12.475232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:12.475200 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:13.042352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:13.042318 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:13.055524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:13.055494 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:13.512381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:13.512353 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:11:16.543082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:16.543039 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:21.293324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:21.293292 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:26.161965 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:26.161923 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:27.782473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:27.782438 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:30.024902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:30.024867 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:11:30.972825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:30.971516 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:35.888709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:35.888667 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:36.188212 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:11:36.188130 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:11:36.258446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.258403 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:11:36.258788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.258756 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" containerID="cri-o://0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712" gracePeriod=30 Nov 02 15:11:36.308947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.308892 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:11:36.309223 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.309176 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" containerID="cri-o://0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952" gracePeriod=30 Nov 02 15:11:36.311135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.311105 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:11:36.311330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:36.311289 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" containerID="cri-o://81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295" gracePeriod=30 Nov 02 15:11:37.060543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:37.060503 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:37.987720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:37.987681 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:40.749569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:40.749521 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:41.603895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.603858 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:11:41.639828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.639781 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:11:41.642902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.642885 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:11:41.656983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.656946 4469 generic.go:356] "Generic (PLEG): container finished" podID="468a249a-d376-4d44-97a5-e53286bfe366" containerID="0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712" exitCode=0 Nov 02 15:11:41.657086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.656991 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" Nov 02 15:11:41.657086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.657027 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" event={"ID":"468a249a-d376-4d44-97a5-e53286bfe366","Type":"ContainerDied","Data":"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712"} Nov 02 15:11:41.657193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.657086 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx" event={"ID":"468a249a-d376-4d44-97a5-e53286bfe366","Type":"ContainerDied","Data":"14177254859df6ea544aac238f4ab66e1b6eab018c708afbcdfbba0f57aec7ca"} Nov 02 15:11:41.657193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.657113 4469 scope.go:122] "RemoveContainer" containerID="0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712" Nov 02 15:11:41.658444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.658421 4469 generic.go:356] "Generic (PLEG): container finished" podID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerID="81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295" exitCode=0 Nov 02 15:11:41.658532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.658463 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" Nov 02 15:11:41.658532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.658507 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" event={"ID":"a5216990-fcd6-4b0c-8689-7c67e677d6b2","Type":"ContainerDied","Data":"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295"} Nov 02 15:11:41.658639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.658563 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx" event={"ID":"a5216990-fcd6-4b0c-8689-7c67e677d6b2","Type":"ContainerDied","Data":"ab9c84dc8c72752be1900a84eef2b573038bf9416e62009a58fffad3a4ce2793"} Nov 02 15:11:41.659858 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.659838 4469 generic.go:356] "Generic (PLEG): container finished" podID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerID="0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952" exitCode=0 Nov 02 15:11:41.659954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.659862 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" event={"ID":"f6609b39-4ce0-4589-9363-34d0618e9be0","Type":"ContainerDied","Data":"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952"} Nov 02 15:11:41.659954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.659886 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" Nov 02 15:11:41.659954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.659907 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6" event={"ID":"f6609b39-4ce0-4589-9363-34d0618e9be0","Type":"ContainerDied","Data":"adeaab528069662395f601419fda77dfaba16eef24f233cd81afb62f4da89a8f"} Nov 02 15:11:41.668279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.668250 4469 scope.go:122] "RemoveContainer" containerID="0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712" Nov 02 15:11:41.668560 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:11:41.668541 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712\": container with ID starting with 0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712 not found: ID does not exist" containerID="0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712" Nov 02 15:11:41.668626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.668566 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712"} err="failed to get container status \"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712\": rpc error: code = NotFound desc = could not find container \"0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712\": container with ID starting with 0a35ea127ccba11afb6108d316750d3b9d76a51e234710b835f53a04d67ed712 not found: ID does not exist" Nov 02 15:11:41.668626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.668591 4469 scope.go:122] "RemoveContainer" containerID="81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295" Nov 02 15:11:41.677169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.677135 4469 scope.go:122] "RemoveContainer" containerID="81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295" Nov 02 15:11:41.677416 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:11:41.677394 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295\": container with ID starting with 81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295 not found: ID does not exist" containerID="81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295" Nov 02 15:11:41.677474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.677425 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295"} err="failed to get container status \"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295\": rpc error: code = NotFound desc = could not find container \"81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295\": container with ID starting with 81a49f002526dff80e36691dffde972c8855a80fa84c2e93b617d1e3688e5295 not found: ID does not exist" Nov 02 15:11:41.677474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.677445 4469 scope.go:122] "RemoveContainer" containerID="0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952" Nov 02 15:11:41.685218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.685202 4469 scope.go:122] "RemoveContainer" containerID="0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952" Nov 02 15:11:41.685471 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:11:41.685450 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952\": container with ID starting with 0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952 not found: ID does not exist" containerID="0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952" Nov 02 15:11:41.685518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.685476 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952"} err="failed to get container status \"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952\": rpc error: code = NotFound desc = could not find container \"0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952\": container with ID starting with 0f8af90f4db6f3de9061c29e782cca74b221697d5576973584ba4422f984c952 not found: ID does not exist" Nov 02 15:11:41.701396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701377 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.701480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701411 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.701480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701431 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.701480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701450 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.701634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701478 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.701634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701520 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.701634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701552 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.701634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701590 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.701871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701675 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.701871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701692 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.701871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701773 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.701871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701832 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.701871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701851 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701870 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701921 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.701957 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702003 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702037 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702040 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.702133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702072 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702148 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702121 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702243 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702310 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702341 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702387 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.702476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702429 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\") pod \"468a249a-d376-4d44-97a5-e53286bfe366\" (UID: \"468a249a-d376-4d44-97a5-e53286bfe366\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702482 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702533 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702569 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702606 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702647 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\") pod \"f6609b39-4ce0-4589-9363-34d0618e9be0\" (UID: \"f6609b39-4ce0-4589-9363-34d0618e9be0\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702707 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702742 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\") pod \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\" (UID: \"a5216990-fcd6-4b0c-8689-7c67e677d6b2\") " Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702531 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702543 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702696 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702868 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.703158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.702998 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:41.703794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.703278 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:41.703794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.703634 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.703794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.703751 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.704135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704111 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704143 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704150 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "kube-api-access-n2w7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.704211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704160 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468a249a-d376-4d44-97a5-e53286bfe366-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704200 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704219 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704234 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6609b39-4ce0-4589-9363-34d0618e9be0-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704250 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704265 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704277 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704292 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704311 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704326 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704341 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704353 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704367 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.704428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704369 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "kube-api-access-pf2x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.705069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.704640 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:11:41.706122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706091 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "kube-api-access-dgs7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.706225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706130 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.706342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706321 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.706404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706383 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.706461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706435 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo" pod "f6609b39-4ce0-4589-9363-34d0618e9be0" (UID: "f6609b39-4ce0-4589-9363-34d0618e9be0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:11:41.706557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706539 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.706765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706750 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:11:41.706854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.706778 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token" pod "a5216990-fcd6-4b0c-8689-7c67e677d6b2" (UID: "a5216990-fcd6-4b0c-8689-7c67e677d6b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:41.707151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.707137 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy" pod "468a249a-d376-4d44-97a5-e53286bfe366" (UID: "468a249a-d376-4d44-97a5-e53286bfe366"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805470 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468a249a-d376-4d44-97a5-e53286bfe366-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805504 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805517 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805530 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805543 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.805556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805555 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468a249a-d376-4d44-97a5-e53286bfe366-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805569 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-kube-api-access-n2w7l\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805582 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805593 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805605 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2x5\" (UniqueName: \"kubernetes.io/projected/468a249a-d376-4d44-97a5-e53286bfe366-kube-api-access-pf2x5\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805620 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6609b39-4ce0-4589-9363-34d0618e9be0-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:41.806112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:41.805633 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgs7h\" (UniqueName: \"kubernetes.io/projected/a5216990-fcd6-4b0c-8689-7c67e677d6b2-kube-api-access-dgs7h\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:11:42.191864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.191750 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:11:42.198598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.198568 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7n2bx"] Nov 02 15:11:42.309549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.309519 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:11:42.315214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.315188 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9znhx"] Nov 02 15:11:42.424998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.424952 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:11:42.426910 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:42.426889 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6rhn6"] Nov 02 15:11:43.123615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:43.123572 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:43.238879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:43.238842 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468a249a-d376-4d44-97a5-e53286bfe366" path="/var/lib/kubelet/pods/468a249a-d376-4d44-97a5-e53286bfe366/volumes" Nov 02 15:11:43.239818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:43.239774 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" path="/var/lib/kubelet/pods/a5216990-fcd6-4b0c-8689-7c67e677d6b2/volumes" Nov 02 15:11:43.240519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:43.240496 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" path="/var/lib/kubelet/pods/f6609b39-4ce0-4589-9363-34d0618e9be0/volumes" Nov 02 15:11:45.653481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:45.653439 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:50.602122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:50.602085 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:55.447702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:55.447660 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:56.369567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:11:56.369538 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:12:00.702398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:00.702328 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:05.553446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:05.553404 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:06.186667 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:12:06.186631 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:12:10.450528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:10.450486 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:15.250539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:15.250507 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:20.079714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:20.079632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:21.641099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641060 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:12:21.641772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641751 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" Nov 02 15:12:21.641892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641775 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" Nov 02 15:12:21.641892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641790 4469 cpu_manager.go:414] "containerMap: removing container" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" Nov 02 15:12:21.641892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641825 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" Nov 02 15:12:21.641892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641852 4469 cpu_manager.go:414] "containerMap: removing container" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" Nov 02 15:12:21.641892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.641860 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" Nov 02 15:12:21.642140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.642000 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5216990-fcd6-4b0c-8689-7c67e677d6b2" containerName="istio-proxy" Nov 02 15:12:21.642140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.642015 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="468a249a-d376-4d44-97a5-e53286bfe366" containerName="istio-proxy" Nov 02 15:12:21.642140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.642027 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6609b39-4ce0-4589-9363-34d0618e9be0" containerName="istio-proxy" Nov 02 15:12:21.645717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.645695 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.691231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.691204 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:12:21.694955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.694936 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:12:21.695068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.695047 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.698619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.698599 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.719259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719236 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719269 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719334 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719370 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdllh\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719403 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719517 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719571 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719635 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.719705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.719693 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.737461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.737435 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6x9j8\"" Nov 02 15:12:21.750867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.750842 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:12:21.762117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.762092 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:12:21.806166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.806131 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:12:21.820591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820561 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.820765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820597 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.820765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820672 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.820765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820705 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.820765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820736 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820776 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820824 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.820859 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821032 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ldw\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821174 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821211 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821239 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821265 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821292 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821330 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821358 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821378 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821428 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821461 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdllh\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821485 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821510 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821548 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9x85\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821594 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821623 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821648 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821667 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.821849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.821695 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.822215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.822060 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.822215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.822060 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.822215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.822132 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.822215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.822159 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.822215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.822196 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.823686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.823661 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.823778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.823727 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.884364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.884327 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.886120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.886092 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdllh\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4qvql\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.923622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923562 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.923622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923609 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.923755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923631 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.923796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923748 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.923882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923821 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x85\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.923882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923873 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.923984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923915 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.923984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923941 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.923984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923973 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.923998 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924020 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924044 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924074 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924117 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924150 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924196 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924238 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924269 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ldw\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924335 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924338 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.924691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924420 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.924691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924119 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.925178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924910 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.925178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.924994 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.925178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.925124 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.925898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.925757 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.926117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.926093 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.926680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.926631 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.927032 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.926986 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.928828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.928228 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.928828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.928674 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.928828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.928680 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.962218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.962178 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:21.986481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.986450 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9x85\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.986682 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.986656 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:21.986767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.986749 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xv4h4\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:21.987009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:21.986988 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ldw\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lwjt6\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:22.008741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.008714 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:22.016559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.015114 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:22.392175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.392149 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:12:22.393702 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:12:22.393673 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce43b306_bf56_4f07_a797_d713f79c618f.slice/crio-ac3b7ef1241ffeab5abf0671243c5b3063ab273aad7285b830977668b39ac77c WatchSource:0}: Error finding container ac3b7ef1241ffeab5abf0671243c5b3063ab273aad7285b830977668b39ac77c: Status 404 returned error can't find the container with id ac3b7ef1241ffeab5abf0671243c5b3063ab273aad7285b830977668b39ac77c Nov 02 15:12:22.396134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.396075 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.396216 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.396178 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.396372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.396220 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.447922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.447899 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:12:22.450750 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.450727 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:12:22.450891 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:12:22.450863 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab4f996_2790_4bea_aeeb_8a062705392a.slice/crio-eec093778115a6d987b43b35ca3a29ef1215168314665a8e48ac3bb261fbf18e WatchSource:0}: Error finding container eec093778115a6d987b43b35ca3a29ef1215168314665a8e48ac3bb261fbf18e: Status 404 returned error can't find the container with id eec093778115a6d987b43b35ca3a29ef1215168314665a8e48ac3bb261fbf18e Nov 02 15:12:22.454063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.453971 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.454063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.454054 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.454208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.454099 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.454670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.454638 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.454732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.454716 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.454780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.454761 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:22.706842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.706740 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:22.898658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.898619 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" event={"ID":"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f","Type":"ContainerStarted","Data":"c9556d808ef776efd5d315648d9fed8e0d308a9b89c27553e374a172e862f2eb"} Nov 02 15:12:22.898867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.898684 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" event={"ID":"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f","Type":"ContainerStarted","Data":"34f171be35af692551746c24d5fef0515e014f7e16eb69e5fee91901b0588d1d"} Nov 02 15:12:22.900558 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.900529 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" event={"ID":"ce43b306-bf56-4f07-a797-d713f79c618f","Type":"ContainerStarted","Data":"090166ec9b2b7f923e2b92fda92d2ddcc6590ecbc916017a3cc3f18e6fba61c7"} Nov 02 15:12:22.900671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.900578 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" event={"ID":"ce43b306-bf56-4f07-a797-d713f79c618f","Type":"ContainerStarted","Data":"ac3b7ef1241ffeab5abf0671243c5b3063ab273aad7285b830977668b39ac77c"} Nov 02 15:12:22.902674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.902651 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" event={"ID":"2ab4f996-2790-4bea-aeeb-8a062705392a","Type":"ContainerStarted","Data":"a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67"} Nov 02 15:12:22.902789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.902695 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" event={"ID":"2ab4f996-2790-4bea-aeeb-8a062705392a","Type":"ContainerStarted","Data":"eec093778115a6d987b43b35ca3a29ef1215168314665a8e48ac3bb261fbf18e"} Nov 02 15:12:22.962466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:22.962427 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:23.009732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.009690 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:23.011112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.011085 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:23.011253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.011140 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:23.018982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.018956 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:23.020426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.020378 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podStartSLOduration=2.020363944 podStartE2EDuration="2.020363944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:23.020016671 +0000 UTC m=+945.903619194" watchObservedRunningTime="2026-11-02 15:12:23.020363944 +0000 UTC m=+945.903966462" Nov 02 15:12:23.020814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.020776 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:23.020899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.020851 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:23.291885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.291005 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podStartSLOduration=2.29098331 podStartE2EDuration="2.29098331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:23.149178467 +0000 UTC m=+946.032780986" watchObservedRunningTime="2026-11-02 15:12:23.29098331 +0000 UTC m=+946.174585830" Nov 02 15:12:23.980824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.973919 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:23.980824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:23.974000 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:24.009231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:24.009189 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:24.009424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:24.009252 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:24.018675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:24.018650 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:24.018795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:24.018687 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:25.065824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.065127 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:25.065824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.065186 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:25.085824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.065766 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:25.085824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.085731 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:25.085824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.075223 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:25.085824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.085791 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:25.180312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.180275 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:25.966661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.966209 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:25.966661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:25.966275 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:26.012177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.012140 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:26.012437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.012406 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:26.024277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.024248 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:26.024464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.024441 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:26.966835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.966206 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:26.966835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:26.966282 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:27.009486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.009455 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:27.009714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.009692 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:27.018626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.018602 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:27.018733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.018637 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:27.966824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.966282 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:27.966824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:27.966353 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:28.009866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.009195 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:28.009866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.009252 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:28.018831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.018783 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:28.018974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.018846 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:28.132369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.132335 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:28.969337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.969298 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:28.969983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:28.969941 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:29.010608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.010568 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:29.010899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.010869 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:29.019316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.019281 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:29.019656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.019626 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:29.963351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.963299 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:12:29.963561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:29.963368 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 15:12:30.010176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:30.010134 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:30.010674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:30.010199 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:30.018614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:30.018578 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:12:30.018745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:30.018642 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 15:12:30.149320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:30.149286 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:31.010138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:31.010098 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:12:31.010337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:31.010162 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 15:12:32.010350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.009939 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4qvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:12:32.010350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.010017 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:12:32.020012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.019976 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:12:32.020206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.020036 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:12:32.075775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.075739 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:32.078642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.078598 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:32.079510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.079484 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:12:32.183680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.183635 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:32.184146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.184076 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:32.185401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.185323 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:12:32.258863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.258757 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podStartSLOduration=11.258720058 podStartE2EDuration="11.258720058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:23.279118216 +0000 UTC m=+946.162720735" watchObservedRunningTime="2026-11-02 15:12:32.258720058 +0000 UTC m=+955.142322577" Nov 02 15:12:32.359410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.359320 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:32.359665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.359640 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:32.360613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:32.360586 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:12:34.959448 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:34.959414 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:36.185509 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:12:36.185468 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:12:36.822972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.822937 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g"] Nov 02 15:12:36.827001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.826978 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.833644 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.833618 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.833759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.833720 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76drd\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-kube-api-access-76drd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.833842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.833771 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.833920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.833903 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f100f692-1c9f-426b-9292-5779f27d4558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.833980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.833941 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.834090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.834000 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f100f692-1c9f-426b-9292-5779f27d4558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.834153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.834111 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.834274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.834247 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.834382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.834319 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.933693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.933659 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g"] Nov 02 15:12:36.934874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.934849 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.934887 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.934917 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76drd\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-kube-api-access-76drd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.934943 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.934991 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f100f692-1c9f-426b-9292-5779f27d4558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935012 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935041 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f100f692-1c9f-426b-9292-5779f27d4558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935122 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935221 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935420 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935466 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935513 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.935740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.935715 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.936160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.936127 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f100f692-1c9f-426b-9292-5779f27d4558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.937877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.937850 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f100f692-1c9f-426b-9292-5779f27d4558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:36.937966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:36.937882 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f100f692-1c9f-426b-9292-5779f27d4558-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:37.008851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.008789 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:37.009026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.008946 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76drd\" (UniqueName: \"kubernetes.io/projected/f100f692-1c9f-426b-9292-5779f27d4558-kube-api-access-76drd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-56p7g\" (UID: \"f100f692-1c9f-426b-9292-5779f27d4558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:37.141090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.141010 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:37.407212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.407135 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:37.574547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.574510 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g"] Nov 02 15:12:37.607108 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:12:37.607078 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf100f692_1c9f_426b_9292_5779f27d4558.slice/crio-07912d609bcca327ecbcbfd5f9b206a196f1ed252b045e45236dde4d34917b4a WatchSource:0}: Error finding container 07912d609bcca327ecbcbfd5f9b206a196f1ed252b045e45236dde4d34917b4a: Status 404 returned error can't find the container with id 07912d609bcca327ecbcbfd5f9b206a196f1ed252b045e45236dde4d34917b4a Nov 02 15:12:37.609682 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.609643 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:37.609774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.609719 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:37.609774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:37.609747 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:38.308097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:38.308057 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" event={"ID":"f100f692-1c9f-426b-9292-5779f27d4558","Type":"ContainerStarted","Data":"1cf4f1707cd84b37642885c561bc1ad0b64b1170bc8fd344c07bd0876e4951d7"} Nov 02 15:12:38.308305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:38.308124 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" event={"ID":"f100f692-1c9f-426b-9292-5779f27d4558","Type":"ContainerStarted","Data":"07912d609bcca327ecbcbfd5f9b206a196f1ed252b045e45236dde4d34917b4a"} Nov 02 15:12:38.350362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:38.350326 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:38.484784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:38.484720 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podStartSLOduration=2.4847000120000002 podStartE2EDuration="2.484700012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:38.483982209 +0000 UTC m=+961.367584740" watchObservedRunningTime="2026-11-02 15:12:38.484700012 +0000 UTC m=+961.368302532" Nov 02 15:12:39.148980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.148936 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:39.151029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.151001 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:39.151173 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.151050 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:39.361584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.361543 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:12:39.366472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.366448 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.449234 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.449152 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"distinct-w0-ba49d0-kserve-self-signed-certs\"" Nov 02 15:12:39.464704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.464674 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.464886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.464745 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.464962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.464914 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.465000 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.464984 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6bz\" (UniqueName: \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.465056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.465018 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.465164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.465144 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.501446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.501418 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:12:39.521463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.521420 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:39.566065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566032 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566097 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566138 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6bz\" (UniqueName: \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566195 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566276 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566351 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566729 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566759 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.566848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.566789 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.568412 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.568383 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.568663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.568645 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.650947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.650911 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6bz\" (UniqueName: \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\") pod \"distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.679297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.679270 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:39.820868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:39.819657 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:40.079271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.079246 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:12:40.081609 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:12:40.081580 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fa67505_1d76_438f_b61d_81b0bd53f704.slice/crio-4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1 WatchSource:0}: Error finding container 4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1: Status 404 returned error can't find the container with id 4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1 Nov 02 15:12:40.142826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.142478 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:40.142826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.142544 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:40.320939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.320906 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" event={"ID":"1fa67505-1d76-438f-b61d-81b0bd53f704","Type":"ContainerStarted","Data":"cd28e19ba6ee284563ae16e43c18e2652b9637e276f091d22afb07bc6b1e8ad8"} Nov 02 15:12:40.321108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.320963 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" event={"ID":"1fa67505-1d76-438f-b61d-81b0bd53f704","Type":"ContainerStarted","Data":"4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1"} Nov 02 15:12:40.440878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.439431 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podStartSLOduration=1.439414488 podStartE2EDuration="1.439414488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:40.439168782 +0000 UTC m=+963.322771302" watchObservedRunningTime="2026-11-02 15:12:40.439414488 +0000 UTC m=+963.323017008" Nov 02 15:12:40.679761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:40.679726 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:41.149876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:41.146265 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:41.149876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:41.146332 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:41.930823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:41.930200 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:42.144824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:42.136144 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:42.146326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:42.145899 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:42.146326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:42.145952 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:43.145215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:43.145176 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:43.145777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:43.145743 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:44.141339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:44.141291 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-56p7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 15:12:44.141541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:44.141359 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" podUID="f100f692-1c9f-426b-9292-5779f27d4558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 15:12:44.773309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:44.773271 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:45.941047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:45.941015 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:45.941670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:45.941641 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:45.942409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:45.942386 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-56p7g" Nov 02 15:12:47.783392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:47.783355 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:12:49.436864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.436825 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:49.639416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.639382 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:12:49.643437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.643415 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.680121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.680095 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:49.692732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.692684 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:49.695757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.695738 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-019efe-openshift-default-dockercfg-9lzm6\"" Nov 02 15:12:49.699246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699225 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699275 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8b8\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699363 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699433 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699467 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699492 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699617 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699664 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.699726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.699713 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.747650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.747609 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:12:49.800485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800450 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800508 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800565 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8b8\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800607 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800663 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800708 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.800793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800740 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800852 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.800914 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.801156 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.801190 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.801321 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.801471 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.801631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.801614 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.803042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.803018 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.803134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.803114 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.861663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.861635 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.861812 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.861743 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8b8\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:49.931172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.931129 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:49.957305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:49.957221 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:50.386672 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.386646 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:12:50.388489 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:12:50.388459 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2312741f_9463_4291_957f_f45ad1ada131.slice/crio-92181ffdfd1cdba6646d0a995fc9dd8c1cf706cf931afd2dc048d56c66250daa WatchSource:0}: Error finding container 92181ffdfd1cdba6646d0a995fc9dd8c1cf706cf931afd2dc048d56c66250daa: Status 404 returned error can't find the container with id 92181ffdfd1cdba6646d0a995fc9dd8c1cf706cf931afd2dc048d56c66250daa Nov 02 15:12:50.390923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.390890 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:50.391005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.390954 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:50.391005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.390982 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:12:50.409261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.409232 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" event={"ID":"2312741f-9463-4291-957f-f45ad1ada131","Type":"ContainerStarted","Data":"92181ffdfd1cdba6646d0a995fc9dd8c1cf706cf931afd2dc048d56c66250daa"} Nov 02 15:12:50.420781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:50.420755 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:12:51.416666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:51.416631 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" event={"ID":"2312741f-9463-4291-957f-f45ad1ada131","Type":"ContainerStarted","Data":"f847f4a1a048a7a8e959e4ebf0fbede26d92b0e5ba2e1dc3735897fe04d98561"} Nov 02 15:12:51.642127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:51.642024 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podStartSLOduration=2.641999882 podStartE2EDuration="2.641999882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:51.586340972 +0000 UTC m=+974.469943495" watchObservedRunningTime="2026-11-02 15:12:51.641999882 +0000 UTC m=+974.525602406" Nov 02 15:12:51.958506 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:51.957777 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:51.959505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:51.959483 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:12:51.959570 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:51.959543 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 15:12:52.958648 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:52.958604 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:12:52.959194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:52.958689 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 15:12:53.958157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:53.958112 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:12:53.958362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:53.958174 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 15:12:54.958559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:54.958522 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:12:54.958986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:54.958585 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 15:12:55.017887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:55.017853 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:55.958192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:55.958153 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:12:55.958446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:55.958235 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 15:12:57.139783 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:57.139726 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:57.140313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:57.140200 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:57.141038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:57.141012 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:12:59.859337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:12:59.859032 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:04.671440 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:04.671403 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:06.186537 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:13:06.186496 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:13:09.589280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:09.589226 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:14.566865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:14.566826 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:19.352121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:19.352076 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:24.153309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:24.153275 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:28.483597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:28.483562 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:28.872176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:28.872094 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:13:28.964649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:28.964617 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:30.831487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.831451 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:13:30.835298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.835281 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.887581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.887558 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"openshift-service-ca.crt\"" Nov 02 15:13:30.901077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.901051 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"kube-root-ca.crt\"" Nov 02 15:13:30.906714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.906684 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.906823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.906719 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.906876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.906849 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.907022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.907001 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxgv\" (UniqueName: \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.907132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.907117 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.907174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.907149 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:30.937308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.937275 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"distinct-w3-42555d-kserve-self-signed-certs\"" Nov 02 15:13:30.937468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.937351 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"default-dockercfg-96rjb\"" Nov 02 15:13:30.940016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.939995 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:13:30.980634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:30.980601 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:31.008232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008197 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008249 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008334 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008414 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxgv\" (UniqueName: \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008499 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008546 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008664 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.008966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.008904 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.009364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.009339 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.010793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.010762 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.011102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.011081 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.069969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.069932 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxgv\" (UniqueName: \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\") pod \"distinct-w3-42555d-kserve-75f67c9f9-56wxg\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.147973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.147902 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:31.537269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.537245 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:13:31.538934 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:13:31.538906 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode406796a_50ff_4cdb_a7be_c90bf453374a.slice/crio-86d09d6858256af757125f8684e42be9583b2367b3ee23b65af9bea54c025c00 WatchSource:0}: Error finding container 86d09d6858256af757125f8684e42be9583b2367b3ee23b65af9bea54c025c00: Status 404 returned error can't find the container with id 86d09d6858256af757125f8684e42be9583b2367b3ee23b65af9bea54c025c00 Nov 02 15:13:31.703880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.703842 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" event={"ID":"e406796a-50ff-4cdb-a7be-c90bf453374a","Type":"ContainerStarted","Data":"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4"} Nov 02 15:13:31.704050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.703891 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" event={"ID":"e406796a-50ff-4cdb-a7be-c90bf453374a","Type":"ContainerStarted","Data":"86d09d6858256af757125f8684e42be9583b2367b3ee23b65af9bea54c025c00"} Nov 02 15:13:31.825626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:31.825510 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podStartSLOduration=1.8254921610000001 podStartE2EDuration="1.825492161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:31.824049357 +0000 UTC m=+1014.707651877" watchObservedRunningTime="2026-11-02 15:13:31.825492161 +0000 UTC m=+1014.709094680" Nov 02 15:13:32.149077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:32.148991 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:33.264568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.264535 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:13:33.268486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.268462 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336093 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336174 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lrq\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336280 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336468 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336564 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336637 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336710 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336759 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.336956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.336784 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.370327 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.370295 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:13:33.437484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437452 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437492 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437513 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437534 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437555 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437611 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.437657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437634 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lrq\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437696 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437837 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.437976 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.438030 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.438146 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.438321 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.438656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.438625 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.440281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.440263 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.440537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.440507 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.498885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.498856 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.502663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.502636 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lrq\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.592117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.592032 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:33.865936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:33.865838 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:34.039581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.039537 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:13:34.045706 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:13:34.045672 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754768b0_cfbc_4a2e_b651_e83c61f8edd4.slice/crio-c9a73fb76c990b85934923374fba64199b912cb6d827ce731a7edb983f88c82a WatchSource:0}: Error finding container c9a73fb76c990b85934923374fba64199b912cb6d827ce731a7edb983f88c82a: Status 404 returned error can't find the container with id c9a73fb76c990b85934923374fba64199b912cb6d827ce731a7edb983f88c82a Nov 02 15:13:34.048246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.048206 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:13:34.048359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.048292 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:13:34.048422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.048339 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:13:34.187168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.187114 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:34.722407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.722370 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" event={"ID":"754768b0-cfbc-4a2e-b651-e83c61f8edd4","Type":"ContainerStarted","Data":"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7"} Nov 02 15:13:34.722925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.722434 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" event={"ID":"754768b0-cfbc-4a2e-b651-e83c61f8edd4","Type":"ContainerStarted","Data":"c9a73fb76c990b85934923374fba64199b912cb6d827ce731a7edb983f88c82a"} Nov 02 15:13:34.897416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:34.897365 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podStartSLOduration=1.8973497099999999 podStartE2EDuration="1.89734971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:34.845887951 +0000 UTC m=+1017.729490473" watchObservedRunningTime="2026-11-02 15:13:34.89734971 +0000 UTC m=+1017.780952230" Nov 02 15:13:35.593149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:35.593111 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:35.594991 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:35.594962 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:13:35.595110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:35.595019 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 15:13:36.186966 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:13:36.186925 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:13:36.592571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:36.592537 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:13:36.592749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:36.592593 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 15:13:37.468821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:37.468265 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:37.592636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:37.592598 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:13:37.592877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:37.592657 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 15:13:37.786128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:37.786045 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:38.592926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:38.592897 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:13:38.593380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:38.593349 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 15:13:38.695692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:38.695651 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:38.901428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:38.901357 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:39.592977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:39.592940 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:13:39.593379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:39.592997 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 15:13:40.786944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:40.786912 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:40.787781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:40.787756 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:40.788898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:40.788872 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:13:41.149056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:41.148969 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:41.186177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:41.186143 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:41.802664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:41.802638 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:13:43.850458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:43.850420 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:48.592093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.592057 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:13:48.597263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.597247 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.700513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.700484 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"default-dockercfg-2g9cz\"" Nov 02 15:13:48.700683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.700569 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"distinct-w1-019efe-kserve-self-signed-certs\"" Nov 02 15:13:48.700683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.700671 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"kube-root-ca.crt\"" Nov 02 15:13:48.703356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.703337 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:13:48.728378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728349 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.728584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728401 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lh6x\" (UniqueName: \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.728584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728553 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.728704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728657 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.728704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728674 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.728704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.728692 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.733652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.733620 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:48.755214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.755190 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"openshift-service-ca.crt\"" Nov 02 15:13:48.829753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.829715 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.829982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.829887 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.829982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.829914 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.829982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.829941 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.830153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.829991 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.830153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.830016 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lh6x\" (UniqueName: \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.830153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.830115 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.830311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.830282 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.830623 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.830599 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.832279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.832251 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.833013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.832997 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.891057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.890981 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lh6x\" (UniqueName: \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\") pod \"distinct-w1-019efe-kserve-65df55b556-5lzm9\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.908989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.908959 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:48.933931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:48.933895 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:49.306835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:49.306782 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:13:49.307677 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:13:49.307649 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6073da20_cbda_46fa_88c6_42deedeab7b0.slice/crio-30ad3487fce3f52ceac993dc5f3aea8640f485ae8786dab00c08ee6749f2a9bc WatchSource:0}: Error finding container 30ad3487fce3f52ceac993dc5f3aea8640f485ae8786dab00c08ee6749f2a9bc: Status 404 returned error can't find the container with id 30ad3487fce3f52ceac993dc5f3aea8640f485ae8786dab00c08ee6749f2a9bc Nov 02 15:13:49.839847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:49.839782 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" event={"ID":"6073da20-cbda-46fa-88c6-42deedeab7b0","Type":"ContainerStarted","Data":"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1"} Nov 02 15:13:49.839847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:49.839848 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" event={"ID":"6073da20-cbda-46fa-88c6-42deedeab7b0","Type":"ContainerStarted","Data":"30ad3487fce3f52ceac993dc5f3aea8640f485ae8786dab00c08ee6749f2a9bc"} Nov 02 15:13:49.909914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:49.909885 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:49.960575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:49.960517 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podStartSLOduration=1.960496883 podStartE2EDuration="1.960496883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:49.9565569 +0000 UTC m=+1032.840159422" watchObservedRunningTime="2026-11-02 15:13:49.960496883 +0000 UTC m=+1032.844099403" Nov 02 15:13:52.335257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:52.335221 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:52.541009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:52.540973 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:53.564205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:53.564172 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:55.679303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:55.679259 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:55.883110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:55.883080 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:58.364609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:58.364572 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:58.909286 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:58.909253 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:58.922095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:58.922070 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:13:59.906733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:13:59.906707 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:14:00.744351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:00.744318 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:03.142990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:03.142950 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:03.656219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:03.656180 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:14:05.643224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.643183 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:14:05.647613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.647592 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.699741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.699714 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c18e79-openshift-default-dockercfg-2fs8l\"" Nov 02 15:14:05.743891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.743856 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.743891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.743893 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744020 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744047 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744093 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744145 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744193 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmx28\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744302 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.744391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.744379 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.753828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.753782 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:14:05.845524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845492 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845544 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845650 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845714 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845730 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845769 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.845839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845790 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845908 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx28\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.845930 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.846020 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.846137 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.846258 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.846429 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.846535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.846491 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.847973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.847943 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.848100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.848083 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.906420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.906363 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.906541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.906442 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx28\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:05.962027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:05.961997 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:06.186730 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:14:06.186643 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:14:06.391917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.391893 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:14:06.393008 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:14:06.392984 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb614d002_216c_4755_acb3_f723b729bb82.slice/crio-e11cdcaaebba1a8a0bdd65d6d2fd17db94a5c1688546a452411a7dfcf3af1ff8 WatchSource:0}: Error finding container e11cdcaaebba1a8a0bdd65d6d2fd17db94a5c1688546a452411a7dfcf3af1ff8: Status 404 returned error can't find the container with id e11cdcaaebba1a8a0bdd65d6d2fd17db94a5c1688546a452411a7dfcf3af1ff8 Nov 02 15:14:06.395304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.395269 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:14:06.395387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.395332 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:14:06.395387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.395382 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:14:06.955873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.955833 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" event={"ID":"b614d002-216c-4755-acb3-f723b729bb82","Type":"ContainerStarted","Data":"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7"} Nov 02 15:14:06.956277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.955927 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" event={"ID":"b614d002-216c-4755-acb3-f723b729bb82","Type":"ContainerStarted","Data":"e11cdcaaebba1a8a0bdd65d6d2fd17db94a5c1688546a452411a7dfcf3af1ff8"} Nov 02 15:14:06.962380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:06.962357 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:07.077922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:07.077869 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podStartSLOduration=2.077854448 podStartE2EDuration="2.077854448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:07.076192092 +0000 UTC m=+1049.959794612" watchObservedRunningTime="2026-11-02 15:14:07.077854448 +0000 UTC m=+1049.961456966" Nov 02 15:14:07.963794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:07.963758 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:07.964267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:07.963831 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:08.165378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:08.163316 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:08.962774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:08.962736 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:08.963002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:08.962844 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:09.963272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:09.963235 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:09.963683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:09.963311 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:10.963083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:10.963047 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:10.963266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:10.963110 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:11.962714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:11.962675 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:11.963165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:11.962732 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:12.545946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:12.545908 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:12.962476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:12.962426 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:14:12.962782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:12.962740 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:14:12.989967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:12.989920 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:13.966734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:13.966708 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:13.967143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:13.966977 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:13.967855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:13.967837 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:14:17.764541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:17.764496 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:22.685848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:22.685772 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:27.589429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:27.589398 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:28.819950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:28.819883 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:32.460819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:32.460758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:34.264763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:34.264715 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:36.189083 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:14:36.189048 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:14:37.351759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:37.351696 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:39.040244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:39.040202 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:42.160958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:42.160910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:45.490337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:45.490295 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:14:47.048033 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:47.047994 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:48.106335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:48.106294 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:51.858094 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:51.858051 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:56.641838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:14:56.641788 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:00.196471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.196438 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:00.200709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.200690 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.210836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.209691 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55lc\" (UniqueName: \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.210836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.209896 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.247885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.247857 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:00.251771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.251754 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.255776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.255751 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:00.259860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.259841 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.308639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.308605 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:00.311508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311482 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75x46\" (UniqueName: \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.311655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311526 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.311655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311606 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.311771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311653 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.311771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311676 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfq2t\" (UniqueName: \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.311909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.311882 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55lc\" (UniqueName: \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.312344 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.312321 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.347632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.347598 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:00.368489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.368457 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:15:00.377259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.377228 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:00.383226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.383194 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55lc\" (UniqueName: \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.413318 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.413288 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75x46\" (UniqueName: \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.413544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.413518 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.413677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.413627 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.413677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.413656 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfq2t\" (UniqueName: \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.414316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.414290 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.414418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.414363 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.475978 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.475949 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfq2t\" (UniqueName: \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.476162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.476140 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75x46\" (UniqueName: \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.520683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.520657 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:00.567159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.567128 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:00.575495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.575468 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:00.903171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.903143 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:00.904318 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:00.904291 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035dedc3_da92_4718_8e0e_29e42ed6e578.slice/crio-4cbea2b91d63de576247a1f805a349fec397112f54585d382e2ac125d98fa016 WatchSource:0}: Error finding container 4cbea2b91d63de576247a1f805a349fec397112f54585d382e2ac125d98fa016: Status 404 returned error can't find the container with id 4cbea2b91d63de576247a1f805a349fec397112f54585d382e2ac125d98fa016 Nov 02 15:15:00.906360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.906344 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:15:00.975120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:00.975093 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:00.975952 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:00.975924 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0693de16_81b6_403d_ae89_396a23a1c30f.slice/crio-a157d366c6ba8aec59c5aad737f982a2a7c94380bf864e2de2d8fd1da63f6b94 WatchSource:0}: Error finding container a157d366c6ba8aec59c5aad737f982a2a7c94380bf864e2de2d8fd1da63f6b94: Status 404 returned error can't find the container with id a157d366c6ba8aec59c5aad737f982a2a7c94380bf864e2de2d8fd1da63f6b94 Nov 02 15:15:01.003873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:01.003849 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:01.007877 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:01.007849 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd340a76f_9f29_4f68_924e_0171c5848332.slice/crio-d89a6606153c664bfa115cc8ffea48d22bbb698bab7a88c62f1a0786a3feb953 WatchSource:0}: Error finding container d89a6606153c664bfa115cc8ffea48d22bbb698bab7a88c62f1a0786a3feb953: Status 404 returned error can't find the container with id d89a6606153c664bfa115cc8ffea48d22bbb698bab7a88c62f1a0786a3feb953 Nov 02 15:15:01.336618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:01.336580 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerStarted","Data":"d89a6606153c664bfa115cc8ffea48d22bbb698bab7a88c62f1a0786a3feb953"} Nov 02 15:15:01.337861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:01.337837 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerStarted","Data":"a157d366c6ba8aec59c5aad737f982a2a7c94380bf864e2de2d8fd1da63f6b94"} Nov 02 15:15:01.339164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:01.339120 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerStarted","Data":"4cbea2b91d63de576247a1f805a349fec397112f54585d382e2ac125d98fa016"} Nov 02 15:15:01.661836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:01.661688 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:05.334610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.334570 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:15:05.349986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.349958 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380234 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380311 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380346 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380396 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380467 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380500 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380563 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380634 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.380684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.380664 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxm4\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.401541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.401513 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:15:05.481765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.481722 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.481973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.481824 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.481973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.481863 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.481973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.481921 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.482134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.482003 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.482134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.482034 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.482134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.482116 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.482288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.482167 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.482288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.482188 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxm4\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.483278 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.483205 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.483442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.483418 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.483494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.483440 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.484347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.484072 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.484923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.484904 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.486435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.486396 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.490643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.490608 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.546639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.546609 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.552651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.552600 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxm4\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:05.670607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:05.670271 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:06.142667 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.142633 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:15:06.148333 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:06.148304 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c49285_8334_4e73_8a3a_9af5076e55cd.slice/crio-bf1942fda7acdfee4979e27106ad399c2066e3bf909e6025313b744269b79e1b WatchSource:0}: Error finding container bf1942fda7acdfee4979e27106ad399c2066e3bf909e6025313b744269b79e1b: Status 404 returned error can't find the container with id bf1942fda7acdfee4979e27106ad399c2066e3bf909e6025313b744269b79e1b Nov 02 15:15:06.158024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.157979 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:06.158114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.158065 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:06.158153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.158111 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:06.200186 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:06.200160 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:15:06.383340 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.383299 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" event={"ID":"f8c49285-8334-4e73-8a3a-9af5076e55cd","Type":"ContainerStarted","Data":"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce"} Nov 02 15:15:06.383860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.383365 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" event={"ID":"f8c49285-8334-4e73-8a3a-9af5076e55cd","Type":"ContainerStarted","Data":"bf1942fda7acdfee4979e27106ad399c2066e3bf909e6025313b744269b79e1b"} Nov 02 15:15:06.507635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.507583 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podStartSLOduration=1.5075622370000001 podStartE2EDuration="1.507562237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:06.504970311 +0000 UTC m=+1109.388572831" watchObservedRunningTime="2026-11-02 15:15:06.507562237 +0000 UTC m=+1109.391164754" Nov 02 15:15:06.560175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.544330 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:06.671583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:06.671503 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:07.390443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.390396 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerStarted","Data":"7726469f5606b3bab88f9eb3da6273432cfe3013558150a90b9a4e0d2ab187d3"} Nov 02 15:15:07.392067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.392040 4469 generic.go:356] "Generic (PLEG): container finished" podID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerID="165a8bc4997f436e679f6457b33b9e8661f2559cbe0170f450f806cf67245b78" exitCode=60 Nov 02 15:15:07.392198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.392127 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerDied","Data":"165a8bc4997f436e679f6457b33b9e8661f2559cbe0170f450f806cf67245b78"} Nov 02 15:15:07.392319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.392303 4469 scope.go:122] "RemoveContainer" containerID="165a8bc4997f436e679f6457b33b9e8661f2559cbe0170f450f806cf67245b78" Nov 02 15:15:07.393597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.393577 4469 generic.go:356] "Generic (PLEG): container finished" podID="d340a76f-9f29-4f68-924e-0171c5848332" containerID="e37c3ca61fe9a8d83cdf491fb75af4a384b172eb8f05d909fa35b7b81a650386" exitCode=60 Nov 02 15:15:07.393678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.393634 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerDied","Data":"e37c3ca61fe9a8d83cdf491fb75af4a384b172eb8f05d909fa35b7b81a650386"} Nov 02 15:15:07.393877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.393863 4469 scope.go:122] "RemoveContainer" containerID="e37c3ca61fe9a8d83cdf491fb75af4a384b172eb8f05d909fa35b7b81a650386" Nov 02 15:15:07.515736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.515648 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" podStartSLOduration=1.969784212 podStartE2EDuration="7.515634187s" totalImagesPullingTime="5.545849975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:07.512539568 +0000 UTC m=+1110.396142087" watchObservedRunningTime="2026-11-02 15:15:07.515634187 +0000 UTC m=+1110.399236706" Nov 02 15:15:07.672358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.672322 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:07.672514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:07.672387 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:08.401487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.401448 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerStarted","Data":"bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799"} Nov 02 15:15:08.403379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.403353 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerStarted","Data":"401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20"} Nov 02 15:15:08.404757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.404732 4469 generic.go:356] "Generic (PLEG): container finished" podID="0693de16-81b6-403d-ae89-396a23a1c30f" containerID="7726469f5606b3bab88f9eb3da6273432cfe3013558150a90b9a4e0d2ab187d3" exitCode=60 Nov 02 15:15:08.404875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.404791 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerDied","Data":"7726469f5606b3bab88f9eb3da6273432cfe3013558150a90b9a4e0d2ab187d3"} Nov 02 15:15:08.405050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.405036 4469 scope.go:122] "RemoveContainer" containerID="7726469f5606b3bab88f9eb3da6273432cfe3013558150a90b9a4e0d2ab187d3" Nov 02 15:15:08.552263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.552192 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" podStartSLOduration=3.323795443 podStartE2EDuration="8.552171974s" totalImagesPullingTime="5.228376531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:08.541568245 +0000 UTC m=+1111.425170779" watchObservedRunningTime="2026-11-02 15:15:08.552171974 +0000 UTC m=+1111.435774494" Nov 02 15:15:08.672121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.672036 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:08.672351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.672324 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:08.783650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:08.783589 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" podStartSLOduration=3.6169057110000002 podStartE2EDuration="8.783574036s" totalImagesPullingTime="5.166668325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:08.780643436 +0000 UTC m=+1111.664245955" watchObservedRunningTime="2026-11-02 15:15:08.783574036 +0000 UTC m=+1111.667176603" Nov 02 15:15:09.413325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.413287 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerStarted","Data":"c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453"} Nov 02 15:15:09.416295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.416272 4469 generic.go:356] "Generic (PLEG): container finished" podID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerID="bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799" exitCode=60 Nov 02 15:15:09.416409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.416353 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerDied","Data":"bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799"} Nov 02 15:15:09.416532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.416432 4469 scope.go:122] "RemoveContainer" containerID="165a8bc4997f436e679f6457b33b9e8661f2559cbe0170f450f806cf67245b78" Nov 02 15:15:09.416591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.416551 4469 scope.go:122] "RemoveContainer" containerID="bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799" Nov 02 15:15:09.416881 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:09.416841 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2_odh-ai-gateway-infra(035dedc3-da92-4718-8e0e-29e42ed6e578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" Nov 02 15:15:09.418528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.418509 4469 generic.go:356] "Generic (PLEG): container finished" podID="d340a76f-9f29-4f68-924e-0171c5848332" containerID="401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20" exitCode=60 Nov 02 15:15:09.418626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.418572 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerDied","Data":"401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20"} Nov 02 15:15:09.418763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.418748 4469 scope.go:122] "RemoveContainer" containerID="401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20" Nov 02 15:15:09.418997 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:09.418978 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb_odh-ai-gateway-infra(d340a76f-9f29-4f68-924e-0171c5848332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" podUID="d340a76f-9f29-4f68-924e-0171c5848332" Nov 02 15:15:09.465707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.465005 4469 scope.go:122] "RemoveContainer" containerID="e37c3ca61fe9a8d83cdf491fb75af4a384b172eb8f05d909fa35b7b81a650386" Nov 02 15:15:09.671473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.671390 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:09.671473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:09.671453 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:10.443835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.443436 4469 scope.go:122] "RemoveContainer" containerID="bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799" Nov 02 15:15:10.443835 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:10.443771 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2_odh-ai-gateway-infra(035dedc3-da92-4718-8e0e-29e42ed6e578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" Nov 02 15:15:10.450249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.450224 4469 scope.go:122] "RemoveContainer" containerID="401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20" Nov 02 15:15:10.450557 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:10.450528 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb_odh-ai-gateway-infra(d340a76f-9f29-4f68-924e-0171c5848332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" podUID="d340a76f-9f29-4f68-924e-0171c5848332" Nov 02 15:15:10.452082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.452053 4469 generic.go:356] "Generic (PLEG): container finished" podID="0693de16-81b6-403d-ae89-396a23a1c30f" containerID="c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453" exitCode=60 Nov 02 15:15:10.452180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.452105 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerDied","Data":"c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453"} Nov 02 15:15:10.452180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.452158 4469 scope.go:122] "RemoveContainer" containerID="7726469f5606b3bab88f9eb3da6273432cfe3013558150a90b9a4e0d2ab187d3" Nov 02 15:15:10.452416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.452391 4469 scope.go:122] "RemoveContainer" containerID="c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453" Nov 02 15:15:10.452683 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:10.452658 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t_odh-ai-gateway-infra(0693de16-81b6-403d-ae89-396a23a1c30f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" Nov 02 15:15:10.671232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.671194 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:10.671384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:10.671251 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:11.366588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:11.366547 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:11.460654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:11.460619 4469 scope.go:122] "RemoveContainer" containerID="c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453" Nov 02 15:15:11.461188 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:11.460917 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t_odh-ai-gateway-infra(0693de16-81b6-403d-ae89-396a23a1c30f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" Nov 02 15:15:11.671419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:11.671330 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:11.671419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:11.671396 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:12.671763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:12.671728 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:15:12.672156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:12.671782 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 15:15:13.676244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:13.676212 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:13.676632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:13.676563 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:13.677238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:13.677216 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:15:14.882177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:14.882144 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:15:14.886531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:14.886504 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:14.991483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:14.991447 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"distinct-2-w3-42555d-kserve-self-signed-certs\"" Nov 02 15:15:14.996786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:14.996751 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:15:15.032042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032007 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwwf\" (UniqueName: \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.032242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032083 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.032242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032173 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.032242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032227 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.032403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032256 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.032453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.032398 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.046065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.046015 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:15.133994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.133911 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwwf\" (UniqueName: \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.133994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.133963 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134029 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134077 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134105 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134251 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134405 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134472 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.134576 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.134560 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.136469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.136443 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.136743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.136725 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.194752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.194724 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwwf\" (UniqueName: \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\") pod \"distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.200662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.200620 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:15.603830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:15.603794 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:15:15.605440 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:15.605413 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac70e2e1_bf7b_4da3_ba9e_a76b86724a5b.slice/crio-a1acccab19cc03ceaaef94a3b86340b2e1f23296bc9ec680a86c445bf2750257 WatchSource:0}: Error finding container a1acccab19cc03ceaaef94a3b86340b2e1f23296bc9ec680a86c445bf2750257: Status 404 returned error can't find the container with id a1acccab19cc03ceaaef94a3b86340b2e1f23296bc9ec680a86c445bf2750257 Nov 02 15:15:16.150872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:16.150772 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:16.493010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:16.492971 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" event={"ID":"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b","Type":"ContainerStarted","Data":"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806"} Nov 02 15:15:16.493175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:16.493018 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" event={"ID":"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b","Type":"ContainerStarted","Data":"a1acccab19cc03ceaaef94a3b86340b2e1f23296bc9ec680a86c445bf2750257"} Nov 02 15:15:16.611149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:16.611094 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podStartSLOduration=2.611078809 podStartE2EDuration="2.611078809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:16.609724843 +0000 UTC m=+1119.493327362" watchObservedRunningTime="2026-11-02 15:15:16.611078809 +0000 UTC m=+1119.494681327" Nov 02 15:15:17.201548 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:17.201520 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:19.232187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:19.232151 4469 scope.go:122] "RemoveContainer" containerID="bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799" Nov 02 15:15:20.232460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:20.232431 4469 scope.go:122] "RemoveContainer" containerID="401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20" Nov 02 15:15:20.541250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:20.541220 4469 generic.go:356] "Generic (PLEG): container finished" podID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerID="5f2dea78745769618c7d2611105ec1147e84d03d413b5764c00815dfae6be3d0" exitCode=60 Nov 02 15:15:20.541385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:20.541366 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerDied","Data":"5f2dea78745769618c7d2611105ec1147e84d03d413b5764c00815dfae6be3d0"} Nov 02 15:15:20.542279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:20.541428 4469 scope.go:122] "RemoveContainer" containerID="bdb4db97f36f2fe091439785c52b8fceac1c3d79dae13d2033e2c7145a694799" Nov 02 15:15:20.542279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:20.541891 4469 scope.go:122] "RemoveContainer" containerID="5f2dea78745769618c7d2611105ec1147e84d03d413b5764c00815dfae6be3d0" Nov 02 15:15:20.542279 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:20.542228 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2_odh-ai-gateway-infra(035dedc3-da92-4718-8e0e-29e42ed6e578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" Nov 02 15:15:21.141746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.141714 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:21.551915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.551876 4469 generic.go:356] "Generic (PLEG): container finished" podID="d340a76f-9f29-4f68-924e-0171c5848332" containerID="52b9604c24d51f950db8da44f4497ade0c00c7416f2deee65f9a29d4d22a6a06" exitCode=60 Nov 02 15:15:21.552404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.551922 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerDied","Data":"52b9604c24d51f950db8da44f4497ade0c00c7416f2deee65f9a29d4d22a6a06"} Nov 02 15:15:21.552404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.551998 4469 scope.go:122] "RemoveContainer" containerID="401c3b62e9fa15cae8aaf93a07db8ddbe388737e7e6df76361974aba72db1b20" Nov 02 15:15:21.552404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.552345 4469 scope.go:122] "RemoveContainer" containerID="52b9604c24d51f950db8da44f4497ade0c00c7416f2deee65f9a29d4d22a6a06" Nov 02 15:15:21.552683 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:21.552654 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb_odh-ai-gateway-infra(d340a76f-9f29-4f68-924e-0171c5848332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" podUID="d340a76f-9f29-4f68-924e-0171c5848332" Nov 02 15:15:21.670151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.669852 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:21.842345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.842315 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:21.925673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.925626 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\" (UniqueName: \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\") pod \"035dedc3-da92-4718-8e0e-29e42ed6e578\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " Nov 02 15:15:21.925898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.925760 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\") pod \"035dedc3-da92-4718-8e0e-29e42ed6e578\" (UID: \"035dedc3-da92-4718-8e0e-29e42ed6e578\") " Nov 02 15:15:21.926138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.926113 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca" pod "035dedc3-da92-4718-8e0e-29e42ed6e578" (UID: "035dedc3-da92-4718-8e0e-29e42ed6e578"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:21.926302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.926287 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035dedc3-da92-4718-8e0e-29e42ed6e578-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:21.928003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:21.927973 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc" pod "035dedc3-da92-4718-8e0e-29e42ed6e578" (UID: "035dedc3-da92-4718-8e0e-29e42ed6e578"). InnerVolumeSpecName "kube-api-access-k55lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:22.027903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.027843 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55lc\" (UniqueName: \"kubernetes.io/projected/035dedc3-da92-4718-8e0e-29e42ed6e578-kube-api-access-k55lc\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:22.560524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.560485 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" Nov 02 15:15:22.561070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.560497 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2" event={"ID":"035dedc3-da92-4718-8e0e-29e42ed6e578","Type":"ContainerDied","Data":"4cbea2b91d63de576247a1f805a349fec397112f54585d382e2ac125d98fa016"} Nov 02 15:15:22.561070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.560615 4469 scope.go:122] "RemoveContainer" containerID="5f2dea78745769618c7d2611105ec1147e84d03d413b5764c00815dfae6be3d0" Nov 02 15:15:22.683163 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.683121 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:22.733468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.733422 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:22.736424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.736394 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893875-gpdt2"] Nov 02 15:15:22.855308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.855273 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:22.941531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.941493 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\") pod \"d340a76f-9f29-4f68-924e-0171c5848332\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " Nov 02 15:15:22.941757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.941688 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\" (UniqueName: \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\") pod \"d340a76f-9f29-4f68-924e-0171c5848332\" (UID: \"d340a76f-9f29-4f68-924e-0171c5848332\") " Nov 02 15:15:22.941990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.941963 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca" pod "d340a76f-9f29-4f68-924e-0171c5848332" (UID: "d340a76f-9f29-4f68-924e-0171c5848332"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:22.942193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.942177 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d340a76f-9f29-4f68-924e-0171c5848332-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:22.943942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:22.943919 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46" pod "d340a76f-9f29-4f68-924e-0171c5848332" (UID: "d340a76f-9f29-4f68-924e-0171c5848332"). InnerVolumeSpecName "kube-api-access-75x46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:23.043405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.043364 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75x46\" (UniqueName: \"kubernetes.io/projected/d340a76f-9f29-4f68-924e-0171c5848332-kube-api-access-75x46\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:23.231744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.231710 4469 scope.go:122] "RemoveContainer" containerID="c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453" Nov 02 15:15:23.241604 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.241568 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" path="/var/lib/kubelet/pods/035dedc3-da92-4718-8e0e-29e42ed6e578/volumes" Nov 02 15:15:23.573939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.573910 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" Nov 02 15:15:23.574458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.573940 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb" event={"ID":"d340a76f-9f29-4f68-924e-0171c5848332","Type":"ContainerDied","Data":"d89a6606153c664bfa115cc8ffea48d22bbb698bab7a88c62f1a0786a3feb953"} Nov 02 15:15:23.574458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.574026 4469 scope.go:122] "RemoveContainer" containerID="52b9604c24d51f950db8da44f4497ade0c00c7416f2deee65f9a29d4d22a6a06" Nov 02 15:15:23.576226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.576124 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerStarted","Data":"cfb0e8f270daea9176415f9a5177fd6c2643e809de125322bba08205dd328472"} Nov 02 15:15:23.858607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.858517 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:23.863233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:23.863201 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893875-qx6lb"] Nov 02 15:15:24.586192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:24.586156 4469 generic.go:356] "Generic (PLEG): container finished" podID="0693de16-81b6-403d-ae89-396a23a1c30f" containerID="cfb0e8f270daea9176415f9a5177fd6c2643e809de125322bba08205dd328472" exitCode=60 Nov 02 15:15:24.586672 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:24.586227 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerDied","Data":"cfb0e8f270daea9176415f9a5177fd6c2643e809de125322bba08205dd328472"} Nov 02 15:15:24.586672 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:24.586295 4469 scope.go:122] "RemoveContainer" containerID="c77e3849f3f0b6cd07d802b70eaa63ba7b354b72bb00d272c2a3804c90a1a453" Nov 02 15:15:24.586812 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:24.586741 4469 scope.go:122] "RemoveContainer" containerID="cfb0e8f270daea9176415f9a5177fd6c2643e809de125322bba08205dd328472" Nov 02 15:15:24.587156 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:24.587128 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t_odh-ai-gateway-infra(0693de16-81b6-403d-ae89-396a23a1c30f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" Nov 02 15:15:24.754836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:24.754785 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:25.055268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.055224 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:15:25.200761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.200727 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:25.214424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.214396 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:25.241031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.241000 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d340a76f-9f29-4f68-924e-0171c5848332" path="/var/lib/kubelet/pods/d340a76f-9f29-4f68-924e-0171c5848332/volumes" Nov 02 15:15:25.609394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.609352 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:15:25.759694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.759657 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:25.888835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.887654 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\") pod \"0693de16-81b6-403d-ae89-396a23a1c30f\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " Nov 02 15:15:25.888835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.887710 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\" (UniqueName: \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\") pod \"0693de16-81b6-403d-ae89-396a23a1c30f\" (UID: \"0693de16-81b6-403d-ae89-396a23a1c30f\") " Nov 02 15:15:25.897450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.897394 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca" pod "0693de16-81b6-403d-ae89-396a23a1c30f" (UID: "0693de16-81b6-403d-ae89-396a23a1c30f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:25.911718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.911127 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t" pod "0693de16-81b6-403d-ae89-396a23a1c30f" (UID: "0693de16-81b6-403d-ae89-396a23a1c30f"). InnerVolumeSpecName "kube-api-access-dfq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:25.946411 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.946353 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:25.992785 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.992735 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0693de16-81b6-403d-ae89-396a23a1c30f-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:25.992785 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:25.992776 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfq2t\" (UniqueName: \"kubernetes.io/projected/0693de16-81b6-403d-ae89-396a23a1c30f-kube-api-access-dfq2t\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:15:26.606112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:26.606074 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" Nov 02 15:15:26.606112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:26.606107 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t" event={"ID":"0693de16-81b6-403d-ae89-396a23a1c30f","Type":"ContainerDied","Data":"a157d366c6ba8aec59c5aad737f982a2a7c94380bf864e2de2d8fd1da63f6b94"} Nov 02 15:15:26.606359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:26.606176 4469 scope.go:122] "RemoveContainer" containerID="cfb0e8f270daea9176415f9a5177fd6c2643e809de125322bba08205dd328472" Nov 02 15:15:26.779484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:26.779438 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:26.782383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:26.782353 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893875-rtn4t"] Nov 02 15:15:27.241427 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:27.241392 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" path="/var/lib/kubelet/pods/0693de16-81b6-403d-ae89-396a23a1c30f/volumes" Nov 02 15:15:30.701514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:30.701476 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:35.506198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:35.506159 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:36.190165 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:15:36.190129 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:15:36.820600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:36.820567 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:37.035843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:37.035811 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:39.219629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:39.219594 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:40.552473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:40.552437 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:44.414080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:44.414037 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:45.459179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:45.459139 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:47.920334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.920299 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:15:47.921221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921199 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921224 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921246 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921253 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921275 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921284 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921302 4469 cpu_manager.go:414] "containerMap: removing container" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921310 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921330 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921337 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921374 4469 cpu_manager.go:414] "containerMap: removing container" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921382 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921392 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921399 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921416 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921423 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921432 4469 cpu_manager.go:414] "containerMap: removing container" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921439 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921586 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921597 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921607 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921615 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921627 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.921814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.921637 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:47.929406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.929382 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974044 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974211 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9t2\" (UniqueName: \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974251 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974268 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974296 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.974451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.974372 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:47.988934 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:47.988890 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:15:48.033949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.033917 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"distinct-2-w5-db3931-kserve-self-signed-certs\"" Nov 02 15:15:48.033949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.033917 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"kube-root-ca.crt\"" Nov 02 15:15:48.034204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.033933 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"openshift-service-ca.crt\"" Nov 02 15:15:48.034204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.034025 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"default-dockercfg-7c6qg\"" Nov 02 15:15:48.075950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.075887 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9t2\" (UniqueName: \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.075950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.075950 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.076196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.075980 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.076196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.076027 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.076196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.076138 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.076354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.076212 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.077127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.077099 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.081159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.080597 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.081159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.080872 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.081159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.081113 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.083353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.083328 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.104403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.104352 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:48.139251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.139220 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9t2\" (UniqueName: \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\") pod \"distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.244894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.244858 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:48.472574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.472536 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:48.623067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.623043 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:15:48.624705 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:48.624674 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb9b2560_ada3_487f_9a7b_0775f9100475.slice/crio-2c8fb71c7150bada9a365c4d7223dbe4bae3429425e32fe1df77b66a7e62f4c7 WatchSource:0}: Error finding container 2c8fb71c7150bada9a365c4d7223dbe4bae3429425e32fe1df77b66a7e62f4c7: Status 404 returned error can't find the container with id 2c8fb71c7150bada9a365c4d7223dbe4bae3429425e32fe1df77b66a7e62f4c7 Nov 02 15:15:48.754788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.754754 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" event={"ID":"db9b2560-ada3-487f-9a7b-0775f9100475","Type":"ContainerStarted","Data":"4d71b0c5af18def826f094b749838fcf20ec0ee52f963b1d7db74550970f5a43"} Nov 02 15:15:48.755009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.754833 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" event={"ID":"db9b2560-ada3-487f-9a7b-0775f9100475","Type":"ContainerStarted","Data":"2c8fb71c7150bada9a365c4d7223dbe4bae3429425e32fe1df77b66a7e62f4c7"} Nov 02 15:15:48.882346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:48.879549 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podStartSLOduration=1.87952764 podStartE2EDuration="1.87952764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:48.871944925 +0000 UTC m=+1151.755547445" watchObservedRunningTime="2026-11-02 15:15:48.87952764 +0000 UTC m=+1151.763130160" Nov 02 15:15:49.245502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:49.245469 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:49.382015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:49.381983 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:50.259740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.259658 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:50.326143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.326115 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:15:50.326827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.326794 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0693de16-81b6-403d-ae89-396a23a1c30f" containerName="cleanup" Nov 02 15:15:50.326870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.326831 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="035dedc3-da92-4718-8e0e-29e42ed6e578" containerName="cleanup" Nov 02 15:15:50.326870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.326841 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d340a76f-9f29-4f68-924e-0171c5848332" containerName="cleanup" Nov 02 15:15:50.330389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.330370 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.382925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.382903 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-42555d-openshift-default-dockercfg-4gl7w\"" Nov 02 15:15:50.404611 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.404587 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.404709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.404647 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.404764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.404737 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.404880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.404852 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.405005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.404985 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.405050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.405033 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.405092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.405080 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.405239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.405215 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzpw\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.405315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.405258 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.435871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.435847 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:15:50.505937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.505910 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.505961 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506019 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506036 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506093 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506147 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzpw\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506174 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506235 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506316 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506757 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506832 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506839 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506875 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.506942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.506908 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.508312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.508292 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.508601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.508583 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.567648 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.567590 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.567841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.567820 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzpw\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:50.644519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:50.644479 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:51.081085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.081058 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:15:51.081691 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:15:51.081664 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c1ebc6_79ba_4e00_886c_e044eec5ecb4.slice/crio-88e466a4e16378ab0862fe9270869f2a0caa284f1713267843c67b223de18028 WatchSource:0}: Error finding container 88e466a4e16378ab0862fe9270869f2a0caa284f1713267843c67b223de18028: Status 404 returned error can't find the container with id 88e466a4e16378ab0862fe9270869f2a0caa284f1713267843c67b223de18028 Nov 02 15:15:51.084207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.084174 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:51.084281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.084241 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:51.084316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.084277 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:15:51.775307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.775269 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" event={"ID":"11c1ebc6-79ba-4e00-886c-e044eec5ecb4","Type":"ContainerStarted","Data":"892d5dc3dd50f01cc1785b5210fd420fafc0bb9a71a3f7d18d8ae8b3f5c91bf2"} Nov 02 15:15:51.775698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.775325 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" event={"ID":"11c1ebc6-79ba-4e00-886c-e044eec5ecb4","Type":"ContainerStarted","Data":"88e466a4e16378ab0862fe9270869f2a0caa284f1713267843c67b223de18028"} Nov 02 15:15:51.912658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:51.912561 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podStartSLOduration=1.9125423270000002 podStartE2EDuration="1.912542327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:51.911601512 +0000 UTC m=+1154.795204030" watchObservedRunningTime="2026-11-02 15:15:51.912542327 +0000 UTC m=+1154.796144848" Nov 02 15:15:52.646824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:52.645147 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:52.648968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:52.648933 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:52.649108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:52.649004 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:53.645824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:53.645172 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:53.645824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:53.645232 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:54.648824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:54.648278 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:54.648824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:54.648345 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:55.150947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:55.149532 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:55.658828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:55.647319 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:55.658828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:55.647396 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:56.647333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:56.647290 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:56.647650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:56.647620 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:57.645776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:57.645732 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:15:57.646258 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:57.645845 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 15:15:58.247401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.247358 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:58.270369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.270331 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:58.651843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.651740 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:58.652293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.652129 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:58.653037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.653014 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:15:58.834018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:58.833985 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:15:59.532578 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:59.532539 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:59.937041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:15:59.936905 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:03.143191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:03.143161 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:16:05.196068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:05.195944 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:06.186533 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:16:06.186502 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:16:10.075305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:10.075258 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:14.858863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:14.858793 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.865934 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:19.865891 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.950581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:19.950535 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:16:19.955867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:19.955841 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:19.998360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:19.998333 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:16:20.002257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.002233 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.019781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.019758 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.019915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.019792 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.019915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.019823 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.019915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.019878 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptsz\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.019951 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020005 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020022 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020046 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020146 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020174 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020200 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020255 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020288 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020318 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020346 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020387 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wv85\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.020473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020417 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.020473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.020441 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.059548 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.059508 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:16:20.109415 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.109378 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:16:20.122109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122030 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122069 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122092 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122181 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptsz\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122257 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122350 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122376 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122414 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122423 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122457 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122484 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122509 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122552 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.122602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122594 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122646 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122675 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122718 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wv85\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122765 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122832 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.123018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.122860 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123151 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.123317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123183 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123306 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123609 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.123713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123689 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.123914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123893 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.124007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.123959 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.124167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.124143 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.130165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.130110 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.130924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.130900 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.136718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.136692 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.139056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.139029 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.185244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.185207 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.185414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.185214 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wv85\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.186255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.186226 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.186544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.186517 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptsz\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.274023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.273981 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:20.317313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.317278 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:20.341235 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.341200 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:16:20.354266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.354242 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.427744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.427472 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29n7\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.427744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.427525 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.427744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.427553 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.427744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.427599 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.428724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.428236 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.428724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.428328 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.428724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.428389 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.428724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.428477 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.428724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.428512 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.451202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.451176 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:16:20.529926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.529898 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29n7\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.529933 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.529953 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.529982 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530105 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530190 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530244 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530324 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530353 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530370 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530491 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.530868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530562 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.531023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.530623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.531266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.531243 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.532605 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.532579 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.533158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.533138 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.591175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.591148 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29n7\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.591601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.591577 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.686543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.686455 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:20.711720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.711696 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:16:20.712208 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:16:20.712183 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa56acb_43d7_41d3_a4af_88174d4118e4.slice/crio-86f9e0b94d74b6658ca229d7c7d37cf79c914e75664a96a868574b7a8095fb65 WatchSource:0}: Error finding container 86f9e0b94d74b6658ca229d7c7d37cf79c914e75664a96a868574b7a8095fb65: Status 404 returned error can't find the container with id 86f9e0b94d74b6658ca229d7c7d37cf79c914e75664a96a868574b7a8095fb65 Nov 02 15:16:20.715020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.714989 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.715106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.715051 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.715106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.715080 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.767540 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:16:20.767493 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7030d72e_a96f_420d_9d3f_a373b196c485.slice/crio-e29b67267dfa0adb8ad2db1b62aaaa6464f8176aca14257afbad776356cf90de WatchSource:0}: Error finding container e29b67267dfa0adb8ad2db1b62aaaa6464f8176aca14257afbad776356cf90de: Status 404 returned error can't find the container with id e29b67267dfa0adb8ad2db1b62aaaa6464f8176aca14257afbad776356cf90de Nov 02 15:16:20.771659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.771159 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:16:20.772459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.772424 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.772552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.772504 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.772614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.772547 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:20.992458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.992422 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" event={"ID":"2fa56acb-43d7-41d3-a4af-88174d4118e4","Type":"ContainerStarted","Data":"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82"} Nov 02 15:16:20.992941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.992477 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" event={"ID":"2fa56acb-43d7-41d3-a4af-88174d4118e4","Type":"ContainerStarted","Data":"86f9e0b94d74b6658ca229d7c7d37cf79c914e75664a96a868574b7a8095fb65"} Nov 02 15:16:20.994311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.994281 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" event={"ID":"7030d72e-a96f-420d-9d3f-a373b196c485","Type":"ContainerStarted","Data":"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de"} Nov 02 15:16:20.994439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:20.994338 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" event={"ID":"7030d72e-a96f-420d-9d3f-a373b196c485","Type":"ContainerStarted","Data":"e29b67267dfa0adb8ad2db1b62aaaa6464f8176aca14257afbad776356cf90de"} Nov 02 15:16:21.075623 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.075596 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:16:21.077981 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:16:21.077958 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273869f1_0187_4321_af11_3ae9bbdfb6a7.slice/crio-93cb968cd9a11bdd89c8a47de3af04dfe1bf6ca61deb12c682cdb1258a0b4e6d WatchSource:0}: Error finding container 93cb968cd9a11bdd89c8a47de3af04dfe1bf6ca61deb12c682cdb1258a0b4e6d: Status 404 returned error can't find the container with id 93cb968cd9a11bdd89c8a47de3af04dfe1bf6ca61deb12c682cdb1258a0b4e6d Nov 02 15:16:21.081472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.081423 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:21.081609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.081585 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:21.081704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.081686 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:21.260874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.246842 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podStartSLOduration=2.246822713 podStartE2EDuration="2.246822713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.118941033 +0000 UTC m=+1184.002543554" watchObservedRunningTime="2026-11-02 15:16:21.246822713 +0000 UTC m=+1184.130425237" Nov 02 15:16:21.266822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.266062 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podStartSLOduration=2.26604181 podStartE2EDuration="2.26604181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.246634049 +0000 UTC m=+1184.130236568" watchObservedRunningTime="2026-11-02 15:16:21.26604181 +0000 UTC m=+1184.149644331" Nov 02 15:16:21.276512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.275115 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:21.276638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.276518 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:21.276638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.276591 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:21.318051 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.318022 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:21.319484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.319389 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:21.319484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:21.319436 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:22.022462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.014513 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" event={"ID":"273869f1-0187-4321-af11-3ae9bbdfb6a7","Type":"ContainerStarted","Data":"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92"} Nov 02 15:16:22.022462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.014580 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" event={"ID":"273869f1-0187-4321-af11-3ae9bbdfb6a7","Type":"ContainerStarted","Data":"93cb968cd9a11bdd89c8a47de3af04dfe1bf6ca61deb12c682cdb1258a0b4e6d"} Nov 02 15:16:22.206825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.202527 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podStartSLOduration=2.202503788 podStartE2EDuration="2.202503788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:22.171330516 +0000 UTC m=+1185.054933088" watchObservedRunningTime="2026-11-02 15:16:22.202503788 +0000 UTC m=+1185.086106309" Nov 02 15:16:22.275793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.275260 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:22.275793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.275342 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:22.318467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.318421 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:22.318669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.318495 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:22.692882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.692384 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:22.692882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.692525 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:22.692882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:22.692569 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:23.275826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.274786 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:23.275826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.274876 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:23.319007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.318969 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:23.319174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.319036 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:23.689883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.689194 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:23.689883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:23.689258 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:24.278248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.278205 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:24.278862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.278828 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:24.319008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.318976 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:24.319250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.319225 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:24.698877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.689377 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:24.698877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.689436 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:24.960883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:24.943224 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:25.314064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.313780 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:25.315635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.315608 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:25.315840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.315814 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:25.407928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.407081 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:25.407928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.407141 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:25.697936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.697601 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:25.698213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:25.698183 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:26.276765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.276729 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:26.277083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.277049 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:26.321251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.321215 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:26.321826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.321776 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:26.691933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.687169 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:26.691933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:26.687233 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:27.320076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.320036 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:27.320292 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.320098 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:27.322251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.322221 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:27.322704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.322283 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:27.687557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.687458 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:27.687557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:27.687506 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:28.275305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.275267 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:16:28.275597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.275567 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 15:16:28.318857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.318793 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:28.319047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.318902 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:28.687177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.687086 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:28.687737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:28.687707 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 15:16:29.280460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.280427 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:29.280961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.280935 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:29.281772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.281746 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:16:29.322502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.322475 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:29.322784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.322765 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:29.323405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:29.323382 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:16:30.729003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.728950 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:16:30.729518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.729016 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:16:30.787067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.787028 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:30.843273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.843240 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:30.843752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.843724 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:30.844652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:30.844630 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:16:35.652002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:35.651947 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:36.187008 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:16:36.186975 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:16:39.515822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:39.515762 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:40.466321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:40.466289 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:45.344912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:45.344874 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:48.809512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:48.809477 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:49.735074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:49.735003 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:50.209559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:50.209530 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:51.533041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.533007 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:16:51.550001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.549968 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.573847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.573817 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.573888 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.573990 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574029 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574156 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574219 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574255 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574383 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.574456 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.574408 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8ng\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.581909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.581886 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:16:51.586207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.586189 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.643289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.643257 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:16:51.676257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676220 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676278 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676346 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676408 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676441 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676508 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.676588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676548 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676647 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676672 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676724 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676795 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xft\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676849 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676873 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.676935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676915 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8ng\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.677322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.676955 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.677322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.677044 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.677322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.677091 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.677322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.677136 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.677322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.677259 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.677577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.677484 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.678239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.678216 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.678391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.678292 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.678909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.678880 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.681904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.681881 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.682078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.682054 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.696445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.696280 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:16:51.742887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.742852 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8ng\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.749174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.749148 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.778588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778551 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xft\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.778765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778597 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.778765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778673 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.778765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778730 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778787 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778863 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778959 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.778980 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.779028 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.779327 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.779409 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.779963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.779913 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.780329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.780301 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.780514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.780486 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.782362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.782334 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.782833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.782780 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.840006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.839924 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.840333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.840309 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xft\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:51.864341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.864312 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:51.899453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:51.899401 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:52.300345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.300308 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:16:52.335892 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:16:52.335862 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3ba8fa_5235_4866_815b_c1d05be06d90.slice/crio-94f4ea748c105d1e19b7a77c0f7d94d10633183d40c61590205483bd0d6a5ff4 WatchSource:0}: Error finding container 94f4ea748c105d1e19b7a77c0f7d94d10633183d40c61590205483bd0d6a5ff4: Status 404 returned error can't find the container with id 94f4ea748c105d1e19b7a77c0f7d94d10633183d40c61590205483bd0d6a5ff4 Nov 02 15:16:52.338700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.338661 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.338827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.338749 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.338827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.338819 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.352074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.352050 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:16:52.354214 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:16:52.354190 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b59b9d_f350_482b_8df2_e081452daefc.slice/crio-d7ba3702861d16a42c8b9cd022818c332514436f9551c8f6b9686995e29f955b WatchSource:0}: Error finding container d7ba3702861d16a42c8b9cd022818c332514436f9551c8f6b9686995e29f955b: Status 404 returned error can't find the container with id d7ba3702861d16a42c8b9cd022818c332514436f9551c8f6b9686995e29f955b Nov 02 15:16:52.357149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.357117 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.357229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.357182 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.357229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.357217 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:16:52.435559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.435515 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" event={"ID":"72b59b9d-f350-482b-8df2-e081452daefc","Type":"ContainerStarted","Data":"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f"} Nov 02 15:16:52.435736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.435586 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" event={"ID":"72b59b9d-f350-482b-8df2-e081452daefc","Type":"ContainerStarted","Data":"d7ba3702861d16a42c8b9cd022818c332514436f9551c8f6b9686995e29f955b"} Nov 02 15:16:52.439981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.439946 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" event={"ID":"aa3ba8fa-5235-4866-815b-c1d05be06d90","Type":"ContainerStarted","Data":"0b9bbd882e3bed662c0d41eaf86de9a6208a3f6e66f8268e95d923caa76c2d2b"} Nov 02 15:16:52.440189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.440162 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" event={"ID":"aa3ba8fa-5235-4866-815b-c1d05be06d90","Type":"ContainerStarted","Data":"94f4ea748c105d1e19b7a77c0f7d94d10633183d40c61590205483bd0d6a5ff4"} Nov 02 15:16:52.561144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.561029 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podStartSLOduration=1.561009475 podStartE2EDuration="1.561009475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:52.558319845 +0000 UTC m=+1215.441922365" watchObservedRunningTime="2026-11-02 15:16:52.561009475 +0000 UTC m=+1215.444611996" Nov 02 15:16:52.680399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.680341 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podStartSLOduration=1.680327343 podStartE2EDuration="1.680327343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:52.678914675 +0000 UTC m=+1215.562517193" watchObservedRunningTime="2026-11-02 15:16:52.680327343 +0000 UTC m=+1215.563929861" Nov 02 15:16:52.865380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.865290 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:16:52.900001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:52.899969 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:16:53.878998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:53.878946 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:53.879629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:53.879573 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:53.901665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:53.901637 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:53.901904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:53.901877 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:54.865259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:54.865219 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:54.865535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:54.865505 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:54.900707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:54.900679 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:54.901205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:54.901174 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:55.147855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:55.147742 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:55.865249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:55.865209 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:55.865502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:55.865472 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:55.900280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:55.900161 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:55.900280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:55.900216 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:56.892122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:56.892083 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:56.892756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:56.892720 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:56.923022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:56.911966 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:56.923022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:56.920475 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:56.923022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:56.920550 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:57.865253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:57.865208 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.865577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:57.865549 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:57.900450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:57.900419 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.900887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:57.900862 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:58.866128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:58.866073 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.866310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:58.866150 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 15:16:58.900061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:58.900026 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.900237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:58.900082 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 15:16:59.229124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:16:59.229086 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:17:00.062231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.062196 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:17:00.063564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.063537 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:17:00.064498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.064471 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:17:00.118132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.118098 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:17:00.118528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.118502 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:17:00.119657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.119635 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:17:00.170005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:00.169978 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:05.590925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:05.590839 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:06.187985 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:17:06.187949 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:17:10.546159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:10.546117 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:15.552539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:15.552502 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:20.129534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.129501 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:17:20.134201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.134179 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.243786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.243751 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:17:20.318553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.318522 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.318743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.318594 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.318743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.318664 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.318946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.318918 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.319077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.318993 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.319145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.319111 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.319210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.319199 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.319335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.319310 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5glv\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.319393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.319348 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.381207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.381062 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:20.421226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421194 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421285 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5glv\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421312 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421366 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421413 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421436 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421514 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421548 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421598 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421625 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.421950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.421890 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.422098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.422075 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.422255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.422227 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.422390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.422319 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.424073 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.424053 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.424208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.424190 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.483419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.483385 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.483552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.483525 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5glv\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.762747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.761272 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:20.834016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.833988 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:17:20.852855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.852051 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:20.904108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:20.904075 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:17:21.033096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033020 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wgv\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033142 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033165 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033243 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033279 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033572 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033366 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033572 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033417 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033572 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033451 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.033572 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.033481 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135357 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wgv\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135502 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135536 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135615 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135656 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.135791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135722 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135833 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135898 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.135933 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.136014 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.136138 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.136170 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.136331 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.136407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.136351 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.138141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.138119 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.138516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.138495 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.194198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.194173 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:17:21.195204 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:17:21.195183 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22984f6a_27d0_4558_bbce_8a4ac8f858f2.slice/crio-9201d7353dd4e2e75abc60e0d798627474c9569691dcd157e658bd1767b90958 WatchSource:0}: Error finding container 9201d7353dd4e2e75abc60e0d798627474c9569691dcd157e658bd1767b90958: Status 404 returned error can't find the container with id 9201d7353dd4e2e75abc60e0d798627474c9569691dcd157e658bd1767b90958 Nov 02 15:17:21.197901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.197868 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:21.198004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.197928 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:21.198004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.197958 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:21.198311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.198290 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.198519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.198500 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wgv\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.474700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.474674 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:21.781882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.776997 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" event={"ID":"22984f6a-27d0-4558-bbce-8a4ac8f858f2","Type":"ContainerStarted","Data":"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c"} Nov 02 15:17:21.781882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.777064 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" event={"ID":"22984f6a-27d0-4558-bbce-8a4ac8f858f2","Type":"ContainerStarted","Data":"9201d7353dd4e2e75abc60e0d798627474c9569691dcd157e658bd1767b90958"} Nov 02 15:17:21.903845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.903778 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podStartSLOduration=1.903765401 podStartE2EDuration="1.903765401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:21.899164637 +0000 UTC m=+1244.782767182" watchObservedRunningTime="2026-11-02 15:17:21.903765401 +0000 UTC m=+1244.787367920" Nov 02 15:17:21.953986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.953948 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:17:21.970192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.970152 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:21.970314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.970247 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:21.970314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:21.970300 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:22.270113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.270079 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:17:22.270610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.270331 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" containerID="cri-o://090166ec9b2b7f923e2b92fda92d2ddcc6590ecbc916017a3cc3f18e6fba61c7" gracePeriod=30 Nov 02 15:17:22.273775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.273752 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:17:22.274213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.274172 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" containerID="cri-o://c9556d808ef776efd5d315648d9fed8e0d308a9b89c27553e374a172e862f2eb" gracePeriod=30 Nov 02 15:17:22.388867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.388764 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:17:22.399172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.398048 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492623 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492708 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492740 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492785 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492838 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492893 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.492991 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.493031 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.494045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.493053 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbd8\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.520884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.520853 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:17:22.574824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.559115 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:17:22.574824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.559176 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:17:22.574824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.559274 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.593965 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594373 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594448 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594487 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqgc\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594530 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594555 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594583 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594625 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594651 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbd8\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594709 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594794 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.594985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594858 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594936 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.594985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594970 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.595127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.594997 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.595127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595049 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.595127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595074 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.595127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595119 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.596156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595632 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.596156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595860 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.596156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.595900 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.596156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.596028 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.598472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.598447 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.600918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.600459 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.606154 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.606128 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.662765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.662682 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.663983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.663961 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbd8\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.696349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696308 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696404 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696507 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696555 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696635 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696706 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696750 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqgc\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696817 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.696873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.696845 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.697124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.697096 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.697857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.697287 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.697857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.697504 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.697857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.697725 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.698121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.697908 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.699034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.699014 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.700995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.700968 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.717946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.717917 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:22.758644 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.758607 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqgc\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.759077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.759052 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:22.761888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.761856 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:22.764171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.763906 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:22.764171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.763974 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:22.805815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.804129 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" event={"ID":"3bb785eb-271a-4042-9225-4417462311a0","Type":"ContainerStarted","Data":"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460"} Nov 02 15:17:22.805815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.804172 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" event={"ID":"3bb785eb-271a-4042-9225-4417462311a0","Type":"ContainerStarted","Data":"205dc6f33f80b8e28834e1a9aea4571ab4183081d5a022d57bab9bf054ad5786"} Nov 02 15:17:22.898577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:22.898555 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:23.017220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.015000 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podStartSLOduration=3.014982278 podStartE2EDuration="3.014982278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:23.012735519 +0000 UTC m=+1245.896338042" watchObservedRunningTime="2026-11-02 15:17:23.014982278 +0000 UTC m=+1245.898584798" Nov 02 15:17:23.295902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.294124 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.295902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.294215 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.295902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.294262 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.295902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.295576 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:17:23.414151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.414117 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:17:23.416380 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:17:23.416344 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce93c0b_c744_4e39_821d_2026121c2233.slice/crio-402e4276a9c3ee524d84d6f19a24262c0ac0370f3038695bc5916bd8faa6ea53 WatchSource:0}: Error finding container 402e4276a9c3ee524d84d6f19a24262c0ac0370f3038695bc5916bd8faa6ea53: Status 404 returned error can't find the container with id 402e4276a9c3ee524d84d6f19a24262c0ac0370f3038695bc5916bd8faa6ea53 Nov 02 15:17:23.420415 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.419773 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.420415 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.419964 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.420415 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.420021 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:23.492336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.492017 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:23.498819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.494929 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:23.498819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.494979 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:23.767828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.767623 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:23.767828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.767681 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:23.815297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.815213 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" event={"ID":"aa0894fd-5c6f-43eb-9209-a27cc095fd59","Type":"ContainerStarted","Data":"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20"} Nov 02 15:17:23.815536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.815515 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" event={"ID":"aa0894fd-5c6f-43eb-9209-a27cc095fd59","Type":"ContainerStarted","Data":"97d57afaf6b98952b45ff8e038e785ef9d0f3c147685bfefd83d11542caec51f"} Nov 02 15:17:23.818031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.817993 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" event={"ID":"9ce93c0b-c744-4e39-821d-2026121c2233","Type":"ContainerStarted","Data":"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae"} Nov 02 15:17:23.818149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.818043 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" event={"ID":"9ce93c0b-c744-4e39-821d-2026121c2233","Type":"ContainerStarted","Data":"402e4276a9c3ee524d84d6f19a24262c0ac0370f3038695bc5916bd8faa6ea53"} Nov 02 15:17:23.900829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.899935 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:23.996297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:23.996226 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podStartSLOduration=1.996202146 podStartE2EDuration="1.996202146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:23.946419734 +0000 UTC m=+1246.830022255" watchObservedRunningTime="2026-11-02 15:17:23.996202146 +0000 UTC m=+1246.879804682" Nov 02 15:17:24.476229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.476189 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:24.476863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.476828 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:24.747279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.747184 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:24.755928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.754927 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:24.780252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.780193 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:24.858900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.856906 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:24.858900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.856971 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:24.900422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.900389 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:24.913482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:24.913442 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:25.625219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.625163 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:17:25.747844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.735767 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:25.747844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.735849 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:25.747844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.738936 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:25.747844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.739003 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:25.747844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.740095 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:17:25.853826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.782052 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:25.891716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.891615 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:25.892007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.812971 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:25.896666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.896624 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:25.923256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.922918 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:25.923256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:25.922983 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:26.034862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.029546 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:26.083825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.074592 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.292852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.270735 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.293979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.293946 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:26.505820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.503014 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:26.505820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.503093 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:26.529821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.514717 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.644881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.644548 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.654023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.653992 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:17:26.740388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.740342 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.740769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.740713 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:26.741236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.741199 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:26.745007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.744983 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:17:26.783603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.779896 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:26.783603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.779955 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:26.784518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.784467 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:26.922912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.906041 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:17:26.923754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.923713 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:17:26.968825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:26.964901 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:17:27.038133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.032577 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:27.063839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.038840 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:27.065316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.065292 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:17:27.137086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.114598 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:27.167277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.167249 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:17:27.169663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.169614 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podStartSLOduration=5.169595354 podStartE2EDuration="5.169595354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:24.095051613 +0000 UTC m=+1246.978654131" watchObservedRunningTime="2026-11-02 15:17:27.169595354 +0000 UTC m=+1250.053197874" Nov 02 15:17:27.221885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.220975 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:27.221885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.221038 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:27.573916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.525451 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 15:17:27.992415 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.537101 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.064944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.064889 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:28.065191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.766897 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:28.090208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.653609 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:28.090705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.950934 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.090894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.090866 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:28.090995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.977117 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.115823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.091103 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:28.203295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.203252 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:28.256877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.815937 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:17:28.282173 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.906787 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.045017 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:27.977265 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.302828 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.233310 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.302948 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:17:28.344002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.302990 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:17:28.406953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.406923 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:17:28.528883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.503916 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.537824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.532267 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:28.684821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.644240 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:28.879202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.724994 4469 generic.go:356] "Generic (PLEG): container finished" podID="ce43b306-bf56-4f07-a797-d713f79c618f" containerID="090166ec9b2b7f923e2b92fda92d2ddcc6590ecbc916017a3cc3f18e6fba61c7" exitCode=0 Nov 02 15:17:28.879750 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.725035 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" event={"ID":"ce43b306-bf56-4f07-a797-d713f79c618f","Type":"ContainerDied","Data":"090166ec9b2b7f923e2b92fda92d2ddcc6590ecbc916017a3cc3f18e6fba61c7"} Nov 02 15:17:28.880361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.814394 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.880542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.880515 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:28.880744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.861686 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.880902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.880879 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:28.968790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.967739 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.968790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:28.967823 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:29.008917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.008862 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:29.026832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.010222 4469 generic.go:356] "Generic (PLEG): container finished" podID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerID="c9556d808ef776efd5d315648d9fed8e0d308a9b89c27553e374a172e862f2eb" exitCode=0 Nov 02 15:17:29.071822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.016990 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" event={"ID":"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f","Type":"ContainerDied","Data":"c9556d808ef776efd5d315648d9fed8e0d308a9b89c27553e374a172e862f2eb"} Nov 02 15:17:29.115977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.090911 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:29.142022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.090942 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:29.243250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.218711 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:17:29.308868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.303360 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:29.308868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.303715 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:17:29.879884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.879453 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:17:29.886111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.886075 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:29.910931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.910885 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:29.977823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.944987 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:29.977823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.945062 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:29.977823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:29.950375 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.007933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.006288 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.007933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.006350 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:30.060823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.015935 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.214233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.032981 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.246942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.224899 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:30.303858 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.147896 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.448145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.147854 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.448145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.247220 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.448145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.401674 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:30.448145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.401730 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.488322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.488287 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:17:30.516823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.489510 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.520823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.518865 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:30.551794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.551760 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.552148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.552109 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.675851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.675821 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.719823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.717983 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:17:30.728177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.728149 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.728521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.728502 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.740267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.740238 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.740514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.740499 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.740683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.740670 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.740792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.740778 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\") pod \"ce43b306-bf56-4f07-a797-d713f79c618f\" (UID: \"ce43b306-bf56-4f07-a797-d713f79c618f\") " Nov 02 15:17:30.754833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.742003 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce43b306-bf56-4f07-a797-d713f79c618f-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:30.761871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.749601 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.771831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.762892 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:30.772749 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:17:30.772705 4469 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.469s" Nov 02 15:17:30.983628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.773139 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:30.983839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.803457 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:30.983982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.761171 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:30.984091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.982365 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.986939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.986876 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:30.987190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.884173 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.851964 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.853049 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.881587 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.986266 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.998319 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.998340 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:30.998422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:30.884140 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:31.006921 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.006890 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.090013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.079818 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "kube-api-access-tdllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:17:31.160851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.138147 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:17:31.215149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.215091 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:31.222451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.222413 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.222620 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.222605 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:31.222773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.222753 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql" event={"ID":"ce43b306-bf56-4f07-a797-d713f79c618f","Type":"ContainerDied","Data":"ac3b7ef1241ffeab5abf0671243c5b3063ab273aad7285b830977668b39ac77c"} Nov 02 15:17:31.222963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.222937 4469 scope.go:122] "RemoveContainer" containerID="090166ec9b2b7f923e2b92fda92d2ddcc6590ecbc916017a3cc3f18e6fba61c7" Nov 02 15:17:31.260871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.242064 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdllh\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-kube-api-access-tdllh\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.419825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.418695 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.419825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.244284 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo" pod "ce43b306-bf56-4f07-a797-d713f79c618f" (UID: "ce43b306-bf56-4f07-a797-d713f79c618f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:17:31.454841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.454758 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:31.455450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.455418 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:17:31.456011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.455981 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:31.496831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.484164 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:31.496831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.484224 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520136 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520217 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520250 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520280 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520312 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.520381 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:17:31.526716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.522859 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:17:31.527968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.527842 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:31.528840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.528817 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:17:31.529629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.529606 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce43b306-bf56-4f07-a797-d713f79c618f-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.529769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.529752 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce43b306-bf56-4f07-a797-d713f79c618f-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.530228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.530210 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce43b306-bf56-4f07-a797-d713f79c618f-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.543988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.543069 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:17:31.553933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.552755 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:17:31.631639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631601 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.631844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631677 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.631844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631731 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.631844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631795 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631930 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.631962 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.632034 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.632124 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.632184 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\") pod \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\" (UID: \"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f\") " Nov 02 15:17:31.632319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.632289 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.633943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.632853 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.633943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.633872 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.634640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.634408 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:17:31.634640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.634447 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.634640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.634554 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.634640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.634611 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.638553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.638531 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "kube-api-access-h7ldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:17:31.643220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.643175 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:17:31.643220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.643175 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:17:31.644555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.644531 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo" pod "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" (UID: "9f4857dc-7efa-4cc3-9d5e-ca2b7846457f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:17:31.722478 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.718830 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:31.722478 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.718892 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736050 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736090 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736104 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736117 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ldw\" (UniqueName: \"kubernetes.io/projected/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-kube-api-access-h7ldw\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736132 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736147 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.737582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.736163 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:17:31.875889 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.873481 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" Nov 02 15:17:31.905529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.905480 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6" event={"ID":"9f4857dc-7efa-4cc3-9d5e-ca2b7846457f","Type":"ContainerDied","Data":"34f171be35af692551746c24d5fef0515e014f7e16eb69e5fee91901b0588d1d"} Nov 02 15:17:31.905782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.905763 4469 scope.go:122] "RemoveContainer" containerID="c9556d808ef776efd5d315648d9fed8e0d308a9b89c27553e374a172e862f2eb" Nov 02 15:17:31.906376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.905843 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:31.906470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:31.906403 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:32.253635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.253593 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:32.487140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.487103 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:32.487825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.487788 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:32.488941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.488913 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:17:32.697922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.697727 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:17:32.708863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.708778 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4qvql"] Nov 02 15:17:32.964883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.952743 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:17:32.977845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:32.977724 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lwjt6"] Nov 02 15:17:33.260612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.260527 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" path="/var/lib/kubelet/pods/9f4857dc-7efa-4cc3-9d5e-ca2b7846457f/volumes" Nov 02 15:17:33.262403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.261964 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" path="/var/lib/kubelet/pods/ce43b306-bf56-4f07-a797-d713f79c618f/volumes" Nov 02 15:17:33.263099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.263079 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:33.265769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.265750 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:33.269421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.269401 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:17:33.315158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.315123 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:33.316851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.316829 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:33.324348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:33.324324 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:17:36.279889 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:17:36.279855 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:17:37.458942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:37.458897 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:37.769400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:37.761342 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.093741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.093608 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:38.146250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.146206 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:17:38.147683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.147652 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.321968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.321924 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:38.449070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.448945 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.460983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.460901 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.461913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.461878 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:17:38.486930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.486889 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.773997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.773936 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:38.806826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:38.774981 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:17:39.164395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.164299 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:17:39.165973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.164624 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.167598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.167577 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:17:39.167768 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.167754 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:17:39.268823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.251021 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.268823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.250018 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:17:39.341946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.341901 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.473444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.473403 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.553847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.519248 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:17:39.555549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.534256 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.937014 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.936908 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:39.990459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.990424 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:17:39.990776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.990755 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:39.991122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.991103 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:17:39.991303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.991289 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:17:39.991469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:39.991436 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:17:40.167044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.166993 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:40.167638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.167606 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:17:40.349974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.349922 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:40.426870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.426813 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:40.554958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.554912 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:40.614641 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.614543 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:40.641872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.634655 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 02 15:17:40.665209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.665164 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:40.775824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:40.760555 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.083840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.067989 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:41.083840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.068056 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.170567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.170525 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.178939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.178884 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.359877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.359443 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:17:41.374621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.374566 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.555961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.555921 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.619445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.619343 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:17:41.622246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.622204 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:41.691386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.691355 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:17:41.726594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.726563 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:17:41.726898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.726879 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:17:41.727299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:41.727280 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:17:45.006734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:45.006681 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:46.810422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.810385 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv"] Nov 02 15:17:46.811160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811141 4469 cpu_manager.go:414] "containerMap: removing container" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" Nov 02 15:17:46.811256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811160 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" Nov 02 15:17:46.811256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811187 4469 cpu_manager.go:414] "containerMap: removing container" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" Nov 02 15:17:46.811256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811195 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" Nov 02 15:17:46.811429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811287 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce43b306-bf56-4f07-a797-d713f79c618f" containerName="istio-proxy" Nov 02 15:17:46.811429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.811298 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4857dc-7efa-4cc3-9d5e-ca2b7846457f" containerName="istio-proxy" Nov 02 15:17:46.816475 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.816447 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.868987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.868954 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 15:17:46.868987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.868955 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 15:17:46.869204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.868996 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 15:17:46.869204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.869043 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2kqpp\"" Nov 02 15:17:46.924382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.924351 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv"] Nov 02 15:17:46.933504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933475 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.933643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933520 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.933763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933726 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.933911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933796 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.933911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933863 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.933911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.933893 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj984\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-kube-api-access-pj984\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:46.934088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:46.934001 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.035901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.035863 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.035901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.035903 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.036166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.035996 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.036166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.036027 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.036166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.036095 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.036166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.036122 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj984\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-kube-api-access-pj984\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.036166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.036156 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.037028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.036993 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.039597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.039537 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.039597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.039542 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.039762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.039589 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.039762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.039751 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.100128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.100054 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj984\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-kube-api-access-pj984\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.100128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.100075 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f055c6fc-cc3f-40df-a429-2a4d6ed481a7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rm9wv\" (UID: \"f055c6fc-cc3f-40df-a429-2a4d6ed481a7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.130772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.130745 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:47.570351 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:17:47.570317 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf055c6fc_cc3f_40df_a429_2a4d6ed481a7.slice/crio-f458862d04951a9638e6d046a05e9f538e73d9b666154f38b56b9f53f5a1b2fa WatchSource:0}: Error finding container f458862d04951a9638e6d046a05e9f538e73d9b666154f38b56b9f53f5a1b2fa: Status 404 returned error can't find the container with id f458862d04951a9638e6d046a05e9f538e73d9b666154f38b56b9f53f5a1b2fa Nov 02 15:17:47.570834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.570794 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv"] Nov 02 15:17:47.619099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:47.619061 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" event={"ID":"f055c6fc-cc3f-40df-a429-2a4d6ed481a7","Type":"ContainerStarted","Data":"f458862d04951a9638e6d046a05e9f538e73d9b666154f38b56b9f53f5a1b2fa"} Nov 02 15:17:49.173424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:49.173388 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:49.874424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:49.874367 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:50.099988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:50.099946 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:51.430854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:51.420366 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:51.430854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:51.420449 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:17:51.661301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:51.661208 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" event={"ID":"f055c6fc-cc3f-40df-a429-2a4d6ed481a7","Type":"ContainerStarted","Data":"396eb77e74c160141f9fd465e38cf6f12f14bfe5b730bfeb14936ac41a892146"} Nov 02 15:17:51.661301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:51.661294 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:51.790184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:51.790132 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" podStartSLOduration=1.944187527 podStartE2EDuration="5.790115014s" totalImagesPullingTime="3.845927487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:51.788332281 +0000 UTC m=+1274.671934801" watchObservedRunningTime="2026-11-02 15:17:51.790115014 +0000 UTC m=+1274.673717532" Nov 02 15:17:52.683486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:52.683451 4469 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rm9wv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:17:52.684007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:52.683507 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" podUID="f055c6fc-cc3f-40df-a429-2a4d6ed481a7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:17:54.817512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:54.817477 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:55.682590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:55.682555 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rm9wv" Nov 02 15:17:59.729220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:17:59.729179 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:04.496153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:04.496099 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:06.189209 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:18:06.189175 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:18:06.791728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.791694 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:18:06.796208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.796186 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870151 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870454 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870561 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870590 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczgs\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870626 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870725 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870861 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.870951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.870894 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.871041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.871026 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.882253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.882168 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:18:06.972426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972389 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.972426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972430 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.972661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972556 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.972725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972687 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.972962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972937 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972986 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.972939 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973023 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973069 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973099 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczgs\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973137 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973146 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973189 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.973971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.973942 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.975390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.975357 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:06.975659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:06.975640 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:07.034780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.034748 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:07.034930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.034862 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczgs\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:07.110333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.110260 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:07.549412 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.549383 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:18:07.551850 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:07.551821 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa05a93f_620a_4649_b108_f2dce51efb36.slice/crio-ab3b6983e978879a2aa6977807c1879fd26ca0c5f2b0cd02631f725ad132646d WatchSource:0}: Error finding container ab3b6983e978879a2aa6977807c1879fd26ca0c5f2b0cd02631f725ad132646d: Status 404 returned error can't find the container with id ab3b6983e978879a2aa6977807c1879fd26ca0c5f2b0cd02631f725ad132646d Nov 02 15:18:07.555930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.555381 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:07.555930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.555469 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:07.555930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.555513 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:07.718898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.718863 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:18:07.719121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.719096 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" containerID="cri-o://a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67" gracePeriod=30 Nov 02 15:18:07.788262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.788205 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" event={"ID":"fa05a93f-620a-4649-b108-f2dce51efb36","Type":"ContainerStarted","Data":"7eed4cbdc2b29deaf3e5666e45e446c00276fa667c3d106024e5350636ecc155"} Nov 02 15:18:07.788431 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.788298 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" event={"ID":"fa05a93f-620a-4649-b108-f2dce51efb36","Type":"ContainerStarted","Data":"ab3b6983e978879a2aa6977807c1879fd26ca0c5f2b0cd02631f725ad132646d"} Nov 02 15:18:07.966931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:07.966838 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podStartSLOduration=1.9667825799999998 podStartE2EDuration="1.96678258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:07.96393778 +0000 UTC m=+1290.847540300" watchObservedRunningTime="2026-11-02 15:18:07.96678258 +0000 UTC m=+1290.850385103" Nov 02 15:18:08.111338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:08.111301 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:08.112664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:08.112635 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:08.112780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:08.112681 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:09.111267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:09.111231 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:09.111651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:09.111288 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:09.583130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:09.583096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:09.956862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:09.956745 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:10.111503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:10.111469 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:10.112018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:10.111528 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:10.165941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:10.165903 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:10.262336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:10.262305 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:18:10.784182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:10.784137 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:11.003665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:11.003632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:11.112353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:11.112265 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:11.112996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:11.112954 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:12.124822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:12.111585 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:12.124822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:12.111643 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:12.390861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:12.390486 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:12.559935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:12.559884 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:12.698476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:12.698380 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.173837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.159194 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:13.173837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.159301 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:13.187076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.178493 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.187076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.180912 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.380835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.375929 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.392882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.390645 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:13.515420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.507480 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.535461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.535421 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 15:18:13.564040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.561910 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:18:13.564040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.562344 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:18:13.618336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.617956 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:13.785822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.778946 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 15:18:13.785822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:13.779368 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:18:14.054906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.048123 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.117347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.113921 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:14.117347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.113987 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:14.185162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.185119 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.186051 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.186017 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:14.186521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.186502 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:18:14.186699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.186685 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:18:14.465228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.395783 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.494824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.468204 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:18:14.562975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.562930 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.564078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.564044 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.566825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.564345 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:18:14.567079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.567063 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:14.567255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.567234 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:18:14.569912 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:18:14.569876 4469 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab4f996_2790_4bea_aeeb_8a062705392a.slice/crio-conmon-a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab4f996_2790_4bea_aeeb_8a062705392a.slice/crio-a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:18:14.583143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.583110 4469 generic.go:356] "Generic (PLEG): container finished" podID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerID="a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67" exitCode=0 Nov 02 15:18:14.596822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.583479 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" event={"ID":"2ab4f996-2790-4bea-aeeb-8a062705392a","Type":"ContainerDied","Data":"a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67"} Nov 02 15:18:14.626852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.620107 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.626852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.620588 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:18:14.784996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.784893 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:14.868958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:14.868901 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" podUID="0aba03b2-70ba-49cb-96ee-896527cd0d76" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.043354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.043254 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:15.043645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.043596 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.101853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.100873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:15.137832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.132107 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:15.138318 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.138184 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:15.189540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.189489 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:15.195029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.194982 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:15.367919 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.367879 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:15.404884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.381743 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.404884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.397848 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:15.478831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.478770 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:15.479117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.479085 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.479769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.479742 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.576882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.573944 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.576882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.575464 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.636824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.622048 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:15.831117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.830984 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 15:18:15.950589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:15.947047 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:18:16.109911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.103262 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" event={"ID":"2ab4f996-2790-4bea-aeeb-8a062705392a","Type":"ContainerDied","Data":"eec093778115a6d987b43b35ca3a29ef1215168314665a8e48ac3bb261fbf18e"} Nov 02 15:18:16.110191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.110163 4469 scope.go:122] "RemoveContainer" containerID="a707e4958e62e7809ba99fa0c868811268aa9c39d10df123470b651c82306f67" Nov 02 15:18:16.113823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.110520 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4" Nov 02 15:18:16.139214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.137989 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:16.139214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.138055 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.139214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.138363 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:16.139214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.138402 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:16.185122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.184977 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:16.185122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.185047 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.206835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.198460 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.206835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.198743 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.206835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.202867 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.206835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.203017 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.236118 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.198295 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:16.237851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.236379 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:18:16.237851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.236468 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.237851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.237486 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.271737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.271678 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:18:16.272232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.272191 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:18:16.291018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.290989 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.295257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.295235 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.333085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.295002 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:18:16.408630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.315963 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:16.408979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.408934 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.409143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.399747 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.412950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.412913 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.663981 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\") pod \"2ab4f996-2790-4bea-aeeb-8a062705392a\" (UID: \"2ab4f996-2790-4bea-aeeb-8a062705392a\") " Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.665372 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.665401 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.665416 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.665431 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab4f996-2790-4bea-aeeb-8a062705392a-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.359039 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "kube-api-access-q9x85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.417088 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.666416 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.668319 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:18:16.680875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.403360 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.705027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.699429 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:18:16.705027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.700886 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy" pod "2ab4f996-2790-4bea-aeeb-8a062705392a" (UID: "2ab4f996-2790-4bea-aeeb-8a062705392a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:18:16.705027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.702072 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.705027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.702995 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.837378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.752904 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.841039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.840992 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:16.927982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.927826 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:16.927982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:16.927902 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:17.157842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.072393 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9x85\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-kube-api-access-q9x85\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:17.158353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.158324 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:17.158497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.158482 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:17.158607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.158592 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:17.158700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.158686 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab4f996-2790-4bea-aeeb-8a062705392a-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:18:17.187984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.187944 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:17.512370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.512328 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:17.513691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.497172 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:17.513984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.511343 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" podUID="0aba03b2-70ba-49cb-96ee-896527cd0d76" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:17.514169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.466098 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:17.707966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.707914 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:17.722006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.721951 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:17.722641 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.722606 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:17.723103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.723082 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:18:17.882874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.876020 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:17.913820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.901444 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:17.913820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:17.901681 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.119943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.115374 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:18.119943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.115442 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:18.140874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.127313 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:18.217024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.216985 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:18:18.408875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.332381 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xv4h4"] Nov 02 15:18:18.457093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.391703 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:18.731829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.731300 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:18.731829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.731361 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.731829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.731465 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.732661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.732631 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:18:18.733710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.733035 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:18.733710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.733607 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:18.753836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.753779 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.848836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.839147 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.848836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.840410 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:18.887833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.887254 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:18.902988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:18.902933 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.134915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.134820 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:19.135194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.135145 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:19.299825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.296666 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.314831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.304433 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:19.527657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.494782 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" path="/var/lib/kubelet/pods/2ab4f996-2790-4bea-aeeb-8a062705392a/volumes" Nov 02 15:18:19.713827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.710283 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:19.739784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.733252 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.740669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.733287 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:18:19.741084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.736855 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.741849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.741831 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:18:19.763168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.761146 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.843886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.842362 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:19.862080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.862047 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:18:19.888029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.887997 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:18:19.911963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.910365 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:19.936827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.927672 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 02 15:18:19.947095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.947066 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:18:19.968100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.967962 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:19.968100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:19.968024 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:20.048450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.048421 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:18:20.194976 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.194898 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:18:20.210956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.195888 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:18:20.215147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.209403 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:20.221825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.216891 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 15:18:20.284972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:20.284934 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:21.586405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:21.582574 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:21.586405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:21.584080 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:21.587050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:21.587024 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:18:22.403726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:22.403687 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:23.328529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:23.328485 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:24.443265 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:24.443229 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:24.679144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:24.679108 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:24.885917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:24.885876 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:25.121781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.121751 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:25.331831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.331769 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:25.815349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.815312 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:18:25.816030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.816014 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" Nov 02 15:18:25.816092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.816031 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" Nov 02 15:18:25.816155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.816145 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4f996-2790-4bea-aeeb-8a062705392a" containerName="istio-proxy" Nov 02 15:18:25.822770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.822746 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.876246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.876212 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5ccdeb27-openshift-default-dockercfg-jr65h\"" Nov 02 15:18:25.890001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.889969 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7splh\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890234 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890214 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890377 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890510 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890559 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890591 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890668 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890700 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.890863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.890787 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.930462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.930128 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:18:25.973383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.973324 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992365 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992421 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992502 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992559 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992704 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992907 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7splh\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.992965 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.993030 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.993892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.993095 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.994743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.994714 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.996208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.996179 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.996772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.996746 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.997264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.997088 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:25.997359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.997301 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.001351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:25.997756 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.014503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.011618 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.059130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.058866 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.059130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.058902 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7splh\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\") pod \"e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.141861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.141773 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:26.191335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.191297 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:26.403728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.403630 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:26.589215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.589186 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:18:26.589505 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:26.589488 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c079aa_3df8_4b44_93bb_bb7e3a7fc4d7.slice/crio-c9d64f45a0febd375c571880d78a85376aa717d05279f42ec9a90cd39aba089c WatchSource:0}: Error finding container c9d64f45a0febd375c571880d78a85376aa717d05279f42ec9a90cd39aba089c: Status 404 returned error can't find the container with id c9d64f45a0febd375c571880d78a85376aa717d05279f42ec9a90cd39aba089c Nov 02 15:18:26.592035 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.592000 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:26.592152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.592078 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:26.592215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:26.592149 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:27.252330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:27.252289 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerStarted","Data":"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de"} Nov 02 15:18:27.252716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:27.252356 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerStarted","Data":"c9d64f45a0febd375c571880d78a85376aa717d05279f42ec9a90cd39aba089c"} Nov 02 15:18:27.382857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:27.382777 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podStartSLOduration=2.382762602 podStartE2EDuration="2.382762602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:27.380305718 +0000 UTC m=+1310.263908237" watchObservedRunningTime="2026-11-02 15:18:27.382762602 +0000 UTC m=+1310.266365120" Nov 02 15:18:27.444977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:27.444949 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:28.089589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.089554 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:28.147076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.147039 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:28.186191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.186159 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:28.260441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.260409 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:28.261091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.261070 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:18:28.319332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.319278 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:28.601038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.600913 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:18:28.617874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.607664 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.662162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.661910 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad0b4b5e\"" Nov 02 15:18:28.662910 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.662879 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad0b4b5e-dockercfg-hsd72\"" Nov 02 15:18:28.665498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.665473 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:18:28.763553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.762095 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96hk\" (UniqueName: \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.763553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.762167 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.766598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.766561 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:18:28.770931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.770907 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:28.829272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.829239 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:18:28.867211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.867122 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrnj\" (UniqueName: \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:28.868023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.867977 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:28.868257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.868238 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96hk\" (UniqueName: \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.868735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.868711 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.870061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.870034 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.930191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.930159 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96hk\" (UniqueName: \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\") pod \"payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:28.970103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.970072 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrnj\" (UniqueName: \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:28.970259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.970172 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:28.970952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:28.970932 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:29.031600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.031566 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrnj\" (UniqueName: \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\") pod \"payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:29.096241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.096207 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:29.230823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.230102 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:29.481552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.481522 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:18:29.482300 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:29.482269 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2410c98_0b74_4298_bf88_07e6bcdabb94.slice/crio-0d151a0753e95cbb2ab05c5d417c957ecaa06b0ed34505fa08043890fdd3223d WatchSource:0}: Error finding container 0d151a0753e95cbb2ab05c5d417c957ecaa06b0ed34505fa08043890fdd3223d: Status 404 returned error can't find the container with id 0d151a0753e95cbb2ab05c5d417c957ecaa06b0ed34505fa08043890fdd3223d Nov 02 15:18:29.512344 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.512320 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:29.622136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.622112 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:18:29.622508 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:29.622486 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b4d2f7_0755_4c72_8b9c_235c82b250fd.slice/crio-f54ca4e19b6517c01f72c6422abbc37f36845063cf880083eb682adb47bbb08f WatchSource:0}: Error finding container f54ca4e19b6517c01f72c6422abbc37f36845063cf880083eb682adb47bbb08f: Status 404 returned error can't find the container with id f54ca4e19b6517c01f72c6422abbc37f36845063cf880083eb682adb47bbb08f Nov 02 15:18:29.717516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:29.717476 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:30.273877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.273838 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" event={"ID":"d2410c98-0b74-4298-bf88-07e6bcdabb94","Type":"ContainerStarted","Data":"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b"} Nov 02 15:18:30.274080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.273905 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" event={"ID":"d2410c98-0b74-4298-bf88-07e6bcdabb94","Type":"ContainerStarted","Data":"0d151a0753e95cbb2ab05c5d417c957ecaa06b0ed34505fa08043890fdd3223d"} Nov 02 15:18:30.274171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.274133 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:30.275692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.275667 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" event={"ID":"c6b4d2f7-0755-4c72-8b9c-235c82b250fd","Type":"ContainerStarted","Data":"15f9e8c4782a6458c4a3aee84b803897fae5acea65836d5117a7508a78492e8e"} Nov 02 15:18:30.275787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.275715 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" event={"ID":"c6b4d2f7-0755-4c72-8b9c-235c82b250fd","Type":"ContainerStarted","Data":"f54ca4e19b6517c01f72c6422abbc37f36845063cf880083eb682adb47bbb08f"} Nov 02 15:18:30.275787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.275738 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:30.391218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.391160 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" podStartSLOduration=2.391141774 podStartE2EDuration="2.391141774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:30.389703721 +0000 UTC m=+1313.273306241" watchObservedRunningTime="2026-11-02 15:18:30.391141774 +0000 UTC m=+1313.274744292" Nov 02 15:18:30.517385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.516396 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" podStartSLOduration=2.516370319 podStartE2EDuration="2.516370319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:30.514097739 +0000 UTC m=+1313.397700258" watchObservedRunningTime="2026-11-02 15:18:30.516370319 +0000 UTC m=+1313.399972888" Nov 02 15:18:30.941621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:30.941584 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:31.151585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:31.151546 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:31.370442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:31.370409 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:33.613256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:33.613217 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:36.186307 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:18:36.186274 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:18:36.713284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.713250 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:18:36.717782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.717758 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.780964 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.780934 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:18:36.786852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.786774 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.786994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.786889 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.786994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.786970 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787050 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787103 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787127 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787199 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787237 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.787335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.787265 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cgj\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888337 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888435 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888503 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888562 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888596 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888642 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888687 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888728 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cgj\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888855 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.888967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888871 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.889077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.888984 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.889133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.889075 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.889248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.889225 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.889413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.889390 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.890970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.890950 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.891059 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.890977 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.949780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.949748 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cgj\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:36.954257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:36.954229 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:37.033897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.033791 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:37.477222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.477194 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:18:37.477552 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:37.477527 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ce43d8_a349_4af9_a379_d4c4cf4d06b1.slice/crio-353d1f23b3c9bcb9a04ddccfd0ca5b0a550b4401ff57905724b10ebd0a4a5a21 WatchSource:0}: Error finding container 353d1f23b3c9bcb9a04ddccfd0ca5b0a550b4401ff57905724b10ebd0a4a5a21: Status 404 returned error can't find the container with id 353d1f23b3c9bcb9a04ddccfd0ca5b0a550b4401ff57905724b10ebd0a4a5a21 Nov 02 15:18:37.480151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.480117 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:37.480247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.480195 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:37.480247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.480238 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:37.930120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:37.930087 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:38.222339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.222279 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:38.382916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.382867 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:18:38.390961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.390933 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.398322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.398284 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" event={"ID":"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1","Type":"ContainerStarted","Data":"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f"} Nov 02 15:18:38.398458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.398348 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" event={"ID":"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1","Type":"ContainerStarted","Data":"353d1f23b3c9bcb9a04ddccfd0ca5b0a550b4401ff57905724b10ebd0a4a5a21"} Nov 02 15:18:38.466450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.466414 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:18:38.479727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.479636 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.480229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.479856 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4988g\" (UniqueName: \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.486256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.486209 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c82fc808-dockercfg-lsmbm\"" Nov 02 15:18:38.486392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.486259 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c82fc808\"" Nov 02 15:18:38.519787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.519750 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:18:38.572824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.572030 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.594774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.589860 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.594774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.589973 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4988g\" (UniqueName: \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.594774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.591210 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.595845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.595787 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:18:38.684096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.682203 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4988g\" (UniqueName: \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\") pod \"payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.694355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.694302 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhm4l\" (UniqueName: \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.694693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.694674 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.710576 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.710546 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:38.796767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.796716 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhm4l\" (UniqueName: \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.796977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.796821 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.798043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.798016 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.813200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.813147 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podStartSLOduration=2.813131276 podStartE2EDuration="2.813131276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:38.68256103 +0000 UTC m=+1321.566163551" watchObservedRunningTime="2026-11-02 15:18:38.813131276 +0000 UTC m=+1321.696733794" Nov 02 15:18:38.885163 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.884015 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhm4l\" (UniqueName: \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\") pod \"payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:38.926824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:38.916071 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:39.078193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.078094 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:39.078969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.078610 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:39.078969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.078662 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:39.387140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.386788 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:18:39.390047 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:39.390019 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1495bbff_0afe_4bab_ad6b_4e4a0a521cd4.slice/crio-94be6d76123442088c5d834a58704ff9b3703c993813c8b1e0031fa7da8318fa WatchSource:0}: Error finding container 94be6d76123442088c5d834a58704ff9b3703c993813c8b1e0031fa7da8318fa: Status 404 returned error can't find the container with id 94be6d76123442088c5d834a58704ff9b3703c993813c8b1e0031fa7da8318fa Nov 02 15:18:39.417608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.417578 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" event={"ID":"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4","Type":"ContainerStarted","Data":"94be6d76123442088c5d834a58704ff9b3703c993813c8b1e0031fa7da8318fa"} Nov 02 15:18:39.469758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.469666 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:39.563932 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.563785 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:18:39.705916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:39.704149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:40.049253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.049221 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:40.049421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.049285 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:40.448879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.446435 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" event={"ID":"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4","Type":"ContainerStarted","Data":"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4"} Nov 02 15:18:40.448879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.447631 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:40.450996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.450970 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" event={"ID":"88cb0a4e-39fc-4361-bf2f-383f36dc6f13","Type":"ContainerStarted","Data":"10da44c6bf2292c5a17091a50d585dca083f05a7e48ffb24d506a9632c47fd9b"} Nov 02 15:18:40.451153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.451126 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" event={"ID":"88cb0a4e-39fc-4361-bf2f-383f36dc6f13","Type":"ContainerStarted","Data":"6c125eb552418e1c177ffbce5c83ce314996bc2d0fc2e96f59febec056075654"} Nov 02 15:18:40.451241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.451163 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:40.520603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.520572 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:40.566265 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.566214 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" podStartSLOduration=2.566197352 podStartE2EDuration="2.566197352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:40.562664836 +0000 UTC m=+1323.446267357" watchObservedRunningTime="2026-11-02 15:18:40.566197352 +0000 UTC m=+1323.449800301" Nov 02 15:18:40.683864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:40.683790 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" podStartSLOduration=2.683774065 podStartE2EDuration="2.683774065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:40.681524037 +0000 UTC m=+1323.565126558" watchObservedRunningTime="2026-11-02 15:18:40.683774065 +0000 UTC m=+1323.567376585" Nov 02 15:18:41.046425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.046383 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:41.046736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.046708 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:41.112587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.112543 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:41.301280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.301081 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:18:41.301905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.301883 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:18:41.362717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.362680 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:41.640892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.634496 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:18:41.655386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.655355 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.710852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.703992 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:18:41.719480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.719448 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5t8\" (UniqueName: \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.719635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.719623 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.722635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.722605 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5ccdeb27-dockercfg-4hs8h\"" Nov 02 15:18:41.722759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.722673 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5ccdeb27\"" Nov 02 15:18:41.827853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.821848 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5t8\" (UniqueName: \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.827853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.822065 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.827853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.823118 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:41.917839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:41.917691 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5t8\" (UniqueName: \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\") pod \"payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:42.044349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:42.044313 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:42.044635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:42.044607 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:42.085441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:42.085407 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:42.763282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:42.763251 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:18:42.765264 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:18:42.765234 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fb9676_9c8d_4be8_9b80_71ec2c6d448e.slice/crio-6b57666b2e6b72cc6b16eeb0273714bb21bb0b3107f32c3dfc4083a59b53b53f WatchSource:0}: Error finding container 6b57666b2e6b72cc6b16eeb0273714bb21bb0b3107f32c3dfc4083a59b53b53f: Status 404 returned error can't find the container with id 6b57666b2e6b72cc6b16eeb0273714bb21bb0b3107f32c3dfc4083a59b53b53f Nov 02 15:18:43.050881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.038826 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:43.050881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.038887 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:43.511662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.511626 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" event={"ID":"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e","Type":"ContainerStarted","Data":"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593"} Nov 02 15:18:43.511842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.511683 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" event={"ID":"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e","Type":"ContainerStarted","Data":"6b57666b2e6b72cc6b16eeb0273714bb21bb0b3107f32c3dfc4083a59b53b53f"} Nov 02 15:18:43.511842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.511753 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:43.630048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:43.629986 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" podStartSLOduration=2.629970171 podStartE2EDuration="2.629970171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:43.628163683 +0000 UTC m=+1326.511766203" watchObservedRunningTime="2026-11-02 15:18:43.629970171 +0000 UTC m=+1326.513572690" Nov 02 15:18:44.034708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:44.034666 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:44.035263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:44.034729 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:45.063149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:45.063111 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:45.063839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:45.063734 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:45.064770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:45.064744 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:18:46.372030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:46.371919 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:50.430381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:50.430345 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:51.565823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:51.564981 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:18:52.031839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.031088 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:18:52.317881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.316077 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 02 15:18:52.492088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.492027 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:52.606879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.605224 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:52.708878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.697930 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:52.747911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.747867 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:18:52.758597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.758549 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:52.819217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.819173 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:52.863878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.860857 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:18:52.863878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.860966 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:18:52.876855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.866230 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:52.941825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:52.940933 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075195 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075282 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075439 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075501 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075537 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvck7\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075602 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075658 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075682 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.098824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.075714 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.249730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.249690 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:53.434577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.401567 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.540703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.513109 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.640326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.640283 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.668012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.667981 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.671927 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.671902 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.696831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.693254 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.733984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.733945 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.748940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.667225 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:53.777997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.777950 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.807935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.806609 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:18:53.807935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.668496 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.807935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.672345 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.807935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.617086 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.809531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.585927 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.702175 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.811190 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.811244 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.811283 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.523312 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.749205 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.812187 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.812226 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvck7\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.785956 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.571017 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.814324 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.814378 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.814574 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:18:53.877417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.849212 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.930945 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.912760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:53.944593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.944009 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:53.969178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.944918 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:18:53.973439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.968434 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvck7\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:53.989053 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:53.989000 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:54.100700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.100619 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:54.126915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.126855 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 02 15:18:54.329422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.329384 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:54.590584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.460074 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:54.590584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.490035 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:54.590584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.560643 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:18:54.780839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.761835 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:54.782912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.782868 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:18:54.878990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.814140 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 15:18:54.880054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.814174 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:54.880435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.814193 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:18:54.885263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.815219 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:54.885610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.834295 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lxpmj" podUID="0aba03b2-70ba-49cb-96ee-896527cd0d76" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:54.970566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.950008 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 02 15:18:54.993856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:54.982325 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:18:55.494474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.494433 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:55.558316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.557920 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:55.890832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.886369 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:55.891733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.891700 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:55.934098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.934055 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:18:55.944185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.944126 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:55.946332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.946304 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 15:18:55.951127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:55.951098 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:56.218874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.204867 4469 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:56.450491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.441570 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:18:56.450491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.441643 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:56.527278 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.527174 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:18:56.528077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.528053 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:56.566487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:56.566448 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" podUID="25f791a3-de1e-4890-943a-ea5230fe6268" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 15:18:57.038013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.037961 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.042412 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.054536 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.054572 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.056220 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.116788 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb" Nov 02 15:18:57.129360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.098229 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 02 15:18:57.163833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.163664 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:18:57.177113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.173311 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:18:57.199149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.199124 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:18:57.199359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.199348 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:18:57.199483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.199475 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:18:57.202219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.202201 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:18:57.465562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.465525 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:18:57.471444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.471400 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:57.471607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.471493 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:57.471607 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.471544 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:18:57.478791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:57.478765 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" event={"ID":"7200a789-6164-4827-b0e2-d5ba74319a92","Type":"ContainerStarted","Data":"6ded0c0261e49bd919b12271fb5b8fa525af816ed534e2b15e1a7f8b73f57532"} Nov 02 15:18:58.499249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:58.499207 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" event={"ID":"7200a789-6164-4827-b0e2-d5ba74319a92","Type":"ContainerStarted","Data":"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec"} Nov 02 15:18:58.670769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:58.670681 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podStartSLOduration=7.670646242 podStartE2EDuration="7.670646242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:58.622574023 +0000 UTC m=+1341.506176544" watchObservedRunningTime="2026-11-02 15:18:58.670646242 +0000 UTC m=+1341.554248764" Nov 02 15:18:59.390586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:59.390554 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:18:59.391670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:59.391639 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:18:59.391832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:59.391729 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 15:18:59.561824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:59.560707 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:59.992662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:18:59.992626 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:00.391521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:00.391429 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:19:00.391521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:00.391496 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 15:19:01.391228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:01.391195 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:19:01.391643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:01.391247 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 15:19:01.944856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:01.944820 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:02.166382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:02.166339 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:02.390935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:02.390900 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:19:02.391103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:02.390961 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 15:19:03.391655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:03.391620 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:19:03.392105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:03.391680 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 15:19:04.787082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:04.787049 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:19:04.788499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:04.788470 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:19:04.790971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:04.790930 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:19:06.192273 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:19:06.192235 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:19:06.867096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:06.867057 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:07.956189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:07.956135 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:08.172341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:08.172287 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:11.882948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:11.882789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:12.306166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:12.306127 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:12.518936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:12.518900 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:12.730679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:12.730644 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:16.740603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:16.740511 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:16.978771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:16.978731 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:18.869932 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:18.869897 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:19.110047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:19.110012 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:19.308017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:19.307984 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:21.712764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:21.712682 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:21.922245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:21.922210 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:22.126430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:22.126347 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:26.509727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:26.509689 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:30.825244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:30.825209 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:19:31.139623 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.139552 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:19:31.144204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.144180 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.209435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.209401 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25v9\" (UniqueName: \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.209867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.209827 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.209951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.209895 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.210013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.210000 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.249380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.249346 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5ccdeb27-serving-cert\"" Nov 02 15:19:31.253525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.253495 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:19:31.311534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.311502 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25v9\" (UniqueName: \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.311689 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.311602 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.311689 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.311634 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.311763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.311701 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.311853 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:19:31.311833 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-serving-cert: secret "maas-api-e2e-derive-5ccdeb27-serving-cert" not found Nov 02 15:19:31.311923 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:19:31.311911 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls podName:000b48f0-6229-4cac-b3d3-48b4bacf52c9 nodeName:}" failed. No retries permitted until 2026-11-02 15:19:31.811896071 +0000 UTC m=+1374.695498574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls") pod "maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" (UID: "000b48f0-6229-4cac-b3d3-48b4bacf52c9") : secret "maas-api-e2e-derive-5ccdeb27-serving-cert" not found Nov 02 15:19:31.312305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.312287 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.314302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.314277 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.375156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.375118 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25v9\" (UniqueName: \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.391522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.391447 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:31.605382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.605337 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:31.837960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.830830 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:31.837960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.832047 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:31.846405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:31.845071 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") pod \"maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:32.060391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:32.060355 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:32.548379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:32.548339 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:19:32.875884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:32.874817 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" event={"ID":"000b48f0-6229-4cac-b3d3-48b4bacf52c9","Type":"ContainerStarted","Data":"e7a228fc5dcbbed6c3346a8d8e951cf06c4969d2c140e6063333c202ea61feeb"} Nov 02 15:19:33.905734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:33.893985 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" event={"ID":"000b48f0-6229-4cac-b3d3-48b4bacf52c9","Type":"ContainerStarted","Data":"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774"} Nov 02 15:19:33.905734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:33.894494 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:34.040977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:34.040915 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" podStartSLOduration=2.1633418620000002 podStartE2EDuration="3.040899105s" totalImagesPullingTime="877.557243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:34.040237656 +0000 UTC m=+1376.923840176" watchObservedRunningTime="2026-11-02 15:19:34.040899105 +0000 UTC m=+1376.924501624" Nov 02 15:19:36.187727 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:19:36.187682 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:19:37.059308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:37.059270 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:39.909788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:39.909757 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:19:40.857090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:40.857019 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:42.089019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:42.088987 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:46.863449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:46.863412 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:49.183223 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:49.183186 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:50.779526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:50.779489 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:51.669241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:51.669185 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:53.701597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.701565 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n"] Nov 02 15:19:53.706088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.706071 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821217 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821299 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821421 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821552 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821592 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b337a30-b105-43bc-820c-0836dfd3dfd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821677 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821765 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5588g\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-kube-api-access-5588g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821916 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.821980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.821953 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.917347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.917313 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n"] Nov 02 15:19:53.923203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923172 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923216 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b337a30-b105-43bc-820c-0836dfd3dfd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923279 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923360 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5588g\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-kube-api-access-5588g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923554 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923469 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923554 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923496 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923642 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923648 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923733 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.923843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923788 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.924022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.923999 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.924527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.924495 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.924686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.924659 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b337a30-b105-43bc-820c-0836dfd3dfd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.924989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.924954 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.927347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.927317 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.927654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.927629 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.987773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.987737 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:53.987947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:53.987840 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5588g\" (UniqueName: \"kubernetes.io/projected/1b337a30-b105-43bc-820c-0836dfd3dfd8-kube-api-access-5588g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6ql9n\" (UID: \"1b337a30-b105-43bc-820c-0836dfd3dfd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:54.020751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:54.020728 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:54.453224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:54.453182 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n"] Nov 02 15:19:54.453374 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:19:54.453350 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b337a30_b105_43bc_820c_0836dfd3dfd8.slice/crio-eee32e5b6f5689c1d83aea81c3a9cb5f18381816ca9cfa6b8844bf80dc63aee5 WatchSource:0}: Error finding container eee32e5b6f5689c1d83aea81c3a9cb5f18381816ca9cfa6b8844bf80dc63aee5: Status 404 returned error can't find the container with id eee32e5b6f5689c1d83aea81c3a9cb5f18381816ca9cfa6b8844bf80dc63aee5 Nov 02 15:19:54.455718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:54.455681 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:19:54.455852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:54.455758 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:19:54.455852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:54.455825 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:19:55.059294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:55.059252 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" event={"ID":"1b337a30-b105-43bc-820c-0836dfd3dfd8","Type":"ContainerStarted","Data":"fb1f2371d0be0e004fec381e353eb24ffca0e9ffd207978ca5a7ac7492f63d1f"} Nov 02 15:19:55.059768 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:55.059316 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" event={"ID":"1b337a30-b105-43bc-820c-0836dfd3dfd8","Type":"ContainerStarted","Data":"eee32e5b6f5689c1d83aea81c3a9cb5f18381816ca9cfa6b8844bf80dc63aee5"} Nov 02 15:19:55.282012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:55.281965 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podStartSLOduration=2.281950684 podStartE2EDuration="2.281950684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:55.230415715 +0000 UTC m=+1398.114018251" watchObservedRunningTime="2026-11-02 15:19:55.281950684 +0000 UTC m=+1398.165553203" Nov 02 15:19:56.021718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:56.021685 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:19:56.023585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:56.023559 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:19:56.023692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:56.023614 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:19:56.694457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:56.692252 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:57.021987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:57.021941 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:19:57.022189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:57.022011 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:19:58.021891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:58.021857 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:19:58.022392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:58.021912 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:19:59.021521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:59.021485 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:19:59.021712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:59.021540 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:19:59.870163 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:19:59.870125 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:00.022019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:00.021982 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:20:00.022197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:00.022038 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:20:01.021980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:01.021941 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:20:01.022377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:01.022007 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:20:01.659240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:01.659203 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:02.022108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:02.022077 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:20:02.022487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:02.022146 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" podUID="1b337a30-b105-43bc-820c-0836dfd3dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 15:20:03.025732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:03.025696 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:20:03.026242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:03.026113 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:20:03.026870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:03.026849 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6ql9n" Nov 02 15:20:06.188180 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:20:06.188136 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:20:06.573841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:06.573791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:08.835028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:08.834993 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb"] Nov 02 15:20:08.840223 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:08.840200 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.010974 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011011 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011124 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011202 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011290 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011251 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011312 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzndj\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-kube-api-access-gzndj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011365 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011414 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.011481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.011471 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.058527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.058488 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb"] Nov 02 15:20:09.112870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.112762 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.112870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.112828 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzndj\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-kube-api-access-gzndj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.112882 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.112939 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113004 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113145 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113179 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113244 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113309 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113372 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113653 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113713 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.113916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.113717 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.115229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.115206 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.115476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.115452 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.174116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.174086 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzndj\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-kube-api-access-gzndj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.175676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.175649 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d428092-3abe-4a4c-a16d-2de2a1c5fe6a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5l5bb\" (UID: \"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.457765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.457667 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:09.898974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.898947 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb"] Nov 02 15:20:09.901325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.901285 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:09.901441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.901364 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:09.901441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:09.901433 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:10.058149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.058112 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:10.175790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.175696 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" event={"ID":"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a","Type":"ContainerStarted","Data":"3646e4b608c32595a72cc37ddc40054f17a0b17ab733885c32337daf795fddbf"} Nov 02 15:20:10.175790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.175772 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" event={"ID":"7d428092-3abe-4a4c-a16d-2de2a1c5fe6a","Type":"ContainerStarted","Data":"57d64455ae67569cb72164406807c99b37a532d2c33fb23340f4625c4eb74277"} Nov 02 15:20:10.305296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.305239 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podStartSLOduration=2.305221146 podStartE2EDuration="2.305221146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:10.301011611 +0000 UTC m=+1413.184614131" watchObservedRunningTime="2026-11-02 15:20:10.305221146 +0000 UTC m=+1413.188823671" Nov 02 15:20:10.458469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.458385 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:10.459708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.459680 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:10.459857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:10.459737 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:11.458591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:11.458558 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:11.459220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:11.459186 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:11.500565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:11.500529 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:12.458526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:12.458487 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:12.458714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:12.458564 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:13.458446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:13.458406 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:13.458631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:13.458467 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:14.458179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:14.458135 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:14.458698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:14.458215 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:15.460848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:15.460763 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:15.461390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:15.460867 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:16.458121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:16.458076 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:16.458302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:16.458151 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:16.861965 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:16.861909 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:17.458942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:17.458899 4469 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:17.459145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:17.458971 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" podUID="7d428092-3abe-4a4c-a16d-2de2a1c5fe6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:18.935305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:18.935271 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:18.939343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:18.939321 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:18.942186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:18.942168 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5l5bb" Nov 02 15:20:22.183950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:22.183908 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:22.799320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:22.799286 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:23.041261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:23.041222 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:28.113981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:28.113929 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:29.302952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:29.302891 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:20:29.302952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:29.302945 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:30.326896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.326862 4469 generic.go:356] "Generic (PLEG): container finished" podID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerID="118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de" exitCode=0 Nov 02 15:20:30.327320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.326930 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerDied","Data":"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de"} Nov 02 15:20:30.327460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.327439 4469 scope.go:122] "RemoveContainer" containerID="118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de" Nov 02 15:20:30.329602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.329562 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:30.329688 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.329658 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:30.329726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:30.329705 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:20:31.358888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:31.358780 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerStarted","Data":"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea"} Nov 02 15:20:32.144824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:32.142629 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:20:32.144824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:32.142758 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.144824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:32.142847 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 15:20:33.146823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:33.145305 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.146823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:33.145382 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 15:20:33.709953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:33.709908 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:34.156323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:34.155913 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:20:34.156323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:34.155979 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 15:20:34.309250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:34.309215 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:20:34.415848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:34.415721 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:20:34.767416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:34.767376 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:35.156050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:35.155953 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.156299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:35.156271 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 15:20:36.149826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:36.148232 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:20:36.149826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:36.148299 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 15:20:36.258107 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:20:36.258050 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:20:36.895360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:36.894936 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:37.114489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.114425 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:37.148542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.148469 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:20:37.148832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.148775 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:20:37.149583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.149564 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:20:37.317029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.316998 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:20:37.441485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.441412 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:20:37.786747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.786710 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:37.993270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:37.993237 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:39.309624 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.309584 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" podUID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" containerName="payload-processing" containerID="cri-o://15f9e8c4782a6458c4a3aee84b803897fae5acea65836d5117a7508a78492e8e" gracePeriod=25 Nov 02 15:20:39.404119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.404075 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:39.416602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.416546 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" podUID="d2410c98-0b74-4298-bf88-07e6bcdabb94" containerName="payload-pre-processing" containerID="cri-o://8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b" gracePeriod=25 Nov 02 15:20:39.474612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.474299 4469 generic.go:356] "Generic (PLEG): container finished" podID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" containerID="15f9e8c4782a6458c4a3aee84b803897fae5acea65836d5117a7508a78492e8e" exitCode=0 Nov 02 15:20:39.474612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.474540 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" event={"ID":"c6b4d2f7-0755-4c72-8b9c-235c82b250fd","Type":"ContainerDied","Data":"15f9e8c4782a6458c4a3aee84b803897fae5acea65836d5117a7508a78492e8e"} Nov 02 15:20:39.649868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.649791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:39.656240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.656211 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:20:39.682868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.680888 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\" (UniqueName: \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\") pod \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " Nov 02 15:20:39.682868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.682187 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume" pod "c6b4d2f7-0755-4c72-8b9c-235c82b250fd" (UID: "c6b4d2f7-0755-4c72-8b9c-235c82b250fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:39.682868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.682234 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\") pod \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\" (UID: \"c6b4d2f7-0755-4c72-8b9c-235c82b250fd\") " Nov 02 15:20:39.683136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.683089 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:39.706040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.706005 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk" pod "c6b4d2f7-0755-4c72-8b9c-235c82b250fd" (UID: "c6b4d2f7-0755-4c72-8b9c-235c82b250fd"). InnerVolumeSpecName "kube-api-access-t96hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:39.780554 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.780526 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:20:39.784594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.784560 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96hk\" (UniqueName: \"kubernetes.io/projected/c6b4d2f7-0755-4c72-8b9c-235c82b250fd-kube-api-access-t96hk\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:39.916481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.916165 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\") pod \"d2410c98-0b74-4298-bf88-07e6bcdabb94\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " Nov 02 15:20:39.916481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.916355 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\" (UniqueName: \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\") pod \"d2410c98-0b74-4298-bf88-07e6bcdabb94\" (UID: \"d2410c98-0b74-4298-bf88-07e6bcdabb94\") " Nov 02 15:20:39.916691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.916579 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume" pod "d2410c98-0b74-4298-bf88-07e6bcdabb94" (UID: "d2410c98-0b74-4298-bf88-07e6bcdabb94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:39.918856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.917451 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2410c98-0b74-4298-bf88-07e6bcdabb94-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:39.928472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:39.928428 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj" pod "d2410c98-0b74-4298-bf88-07e6bcdabb94" (UID: "d2410c98-0b74-4298-bf88-07e6bcdabb94"). InnerVolumeSpecName "kube-api-access-hhrnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:40.019193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.019161 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrnj\" (UniqueName: \"kubernetes.io/projected/d2410c98-0b74-4298-bf88-07e6bcdabb94-kube-api-access-hhrnj\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:40.484007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.483970 4469 generic.go:356] "Generic (PLEG): container finished" podID="d2410c98-0b74-4298-bf88-07e6bcdabb94" containerID="8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b" exitCode=0 Nov 02 15:20:40.484474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.484016 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" Nov 02 15:20:40.484474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.484063 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" event={"ID":"d2410c98-0b74-4298-bf88-07e6bcdabb94","Type":"ContainerDied","Data":"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b"} Nov 02 15:20:40.484474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.484124 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn" event={"ID":"d2410c98-0b74-4298-bf88-07e6bcdabb94","Type":"ContainerDied","Data":"0d151a0753e95cbb2ab05c5d417c957ecaa06b0ed34505fa08043890fdd3223d"} Nov 02 15:20:40.484474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.484150 4469 scope.go:122] "RemoveContainer" containerID="8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b" Nov 02 15:20:40.486274 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.486249 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" event={"ID":"c6b4d2f7-0755-4c72-8b9c-235c82b250fd","Type":"ContainerDied","Data":"f54ca4e19b6517c01f72c6422abbc37f36845063cf880083eb682adb47bbb08f"} Nov 02 15:20:40.486402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.486342 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq" Nov 02 15:20:40.497608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.497512 4469 scope.go:122] "RemoveContainer" containerID="8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b" Nov 02 15:20:40.498039 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:20:40.498015 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b\": container with ID starting with 8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b not found: ID does not exist" containerID="8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b" Nov 02 15:20:40.498130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.498043 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b"} err="failed to get container status \"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b\": rpc error: code = NotFound desc = could not find container \"8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b\": container with ID starting with 8d4985dbfe3d9bbabeccb9ff5a6a277519cd95e0c2f066e200c01c0e3bdb890b not found: ID does not exist" Nov 02 15:20:40.498130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.498062 4469 scope.go:122] "RemoveContainer" containerID="15f9e8c4782a6458c4a3aee84b803897fae5acea65836d5117a7508a78492e8e" Nov 02 15:20:40.661813 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.661729 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:20:40.711017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.710988 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0b4b5e-76f7f4795f-nkmnn"] Nov 02 15:20:40.787183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.787149 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:20:40.803970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:40.803943 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0b4b5e-5b66d56bdf-l2twq"] Nov 02 15:20:41.217577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:41.217541 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:41.248862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:41.248821 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" path="/var/lib/kubelet/pods/c6b4d2f7-0755-4c72-8b9c-235c82b250fd/volumes" Nov 02 15:20:41.249362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:41.249337 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2410c98-0b74-4298-bf88-07e6bcdabb94" path="/var/lib/kubelet/pods/d2410c98-0b74-4298-bf88-07e6bcdabb94/volumes" Nov 02 15:20:42.317518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.317455 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" podUID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" containerName="payload-processing" containerID="cri-o://10da44c6bf2292c5a17091a50d585dca083f05a7e48ffb24d506a9632c47fd9b" gracePeriod=25 Nov 02 15:20:42.441810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.441738 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerName="payload-pre-processing" containerID="cri-o://83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4" gracePeriod=25 Nov 02 15:20:42.518684 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:20:42.518649 4469 logging.go:55] [core] [Channel #2819 SubChannel #2820]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.87:50052", ServerName: "10.134.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.87:50052: connect: connection refused" Nov 02 15:20:42.523622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.523572 4469 generic.go:356] "Generic (PLEG): container finished" podID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" containerID="10da44c6bf2292c5a17091a50d585dca083f05a7e48ffb24d506a9632c47fd9b" exitCode=0 Nov 02 15:20:42.523792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.523769 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" event={"ID":"88cb0a4e-39fc-4361-bf2f-383f36dc6f13","Type":"ContainerDied","Data":"10da44c6bf2292c5a17091a50d585dca083f05a7e48ffb24d506a9632c47fd9b"} Nov 02 15:20:42.630981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.630572 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:20:42.713555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.713529 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:20:42.771339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.771280 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\") pod \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " Nov 02 15:20:42.771521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.771372 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\" (UniqueName: \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\") pod \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " Nov 02 15:20:42.771651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.771632 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\" (UniqueName: \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\") pod \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\" (UID: \"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4\") " Nov 02 15:20:42.771724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.771679 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\") pod \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\" (UID: \"88cb0a4e-39fc-4361-bf2f-383f36dc6f13\") " Nov 02 15:20:42.771786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.771726 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume" pod "1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" (UID: "1495bbff-0afe-4bab-ad6b-4e4a0a521cd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:42.772302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.772275 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume" pod "88cb0a4e-39fc-4361-bf2f-383f36dc6f13" (UID: "88cb0a4e-39fc-4361-bf2f-383f36dc6f13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:42.772550 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.772529 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:42.772630 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.772557 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:42.773756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.773731 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l" pod "88cb0a4e-39fc-4361-bf2f-383f36dc6f13" (UID: "88cb0a4e-39fc-4361-bf2f-383f36dc6f13"). InnerVolumeSpecName "kube-api-access-qhm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:42.774261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.774230 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g" pod "1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" (UID: "1495bbff-0afe-4bab-ad6b-4e4a0a521cd4"). InnerVolumeSpecName "kube-api-access-4988g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:42.874657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.874573 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4988g\" (UniqueName: \"kubernetes.io/projected/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4-kube-api-access-4988g\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:42.874657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.874608 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhm4l\" (UniqueName: \"kubernetes.io/projected/88cb0a4e-39fc-4361-bf2f-383f36dc6f13-kube-api-access-qhm4l\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:42.939513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:42.939478 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:43.519347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.519261 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.87:50052\" within 1s: context deadline exceeded" Nov 02 15:20:43.531726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.531699 4469 generic.go:356] "Generic (PLEG): container finished" podID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerID="83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4" exitCode=0 Nov 02 15:20:43.531863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.531741 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" Nov 02 15:20:43.531863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.531782 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" event={"ID":"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4","Type":"ContainerDied","Data":"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4"} Nov 02 15:20:43.531863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.531847 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p" event={"ID":"1495bbff-0afe-4bab-ad6b-4e4a0a521cd4","Type":"ContainerDied","Data":"94be6d76123442088c5d834a58704ff9b3703c993813c8b1e0031fa7da8318fa"} Nov 02 15:20:43.532013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.531871 4469 scope.go:122] "RemoveContainer" containerID="83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4" Nov 02 15:20:43.533410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.533389 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" Nov 02 15:20:43.533525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.533404 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86" event={"ID":"88cb0a4e-39fc-4361-bf2f-383f36dc6f13","Type":"ContainerDied","Data":"6c125eb552418e1c177ffbce5c83ce314996bc2d0fc2e96f59febec056075654"} Nov 02 15:20:43.542107 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.542091 4469 scope.go:122] "RemoveContainer" containerID="83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4" Nov 02 15:20:43.542369 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:20:43.542350 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4\": container with ID starting with 83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4 not found: ID does not exist" containerID="83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4" Nov 02 15:20:43.542432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.542376 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4"} err="failed to get container status \"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4\": rpc error: code = NotFound desc = could not find container \"83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4\": container with ID starting with 83ef4ff3ba3ade552d2d3db09140f21a7e0e3a4c9b57d01146dd75a50fc00bf4 not found: ID does not exist" Nov 02 15:20:43.542432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.542394 4469 scope.go:122] "RemoveContainer" containerID="10da44c6bf2292c5a17091a50d585dca083f05a7e48ffb24d506a9632c47fd9b" Nov 02 15:20:43.719251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.719220 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:20:43.767877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.767793 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c82fc808-559f785f49-gcs86"] Nov 02 15:20:43.830281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.830212 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:20:43.837866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.837841 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c82fc808-759c665f7f-79x9p"] Nov 02 15:20:43.885437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:43.885405 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:20:44.304450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:44.304418 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:44.528521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:44.528485 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:45.240320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:45.240286 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" path="/var/lib/kubelet/pods/1495bbff-0afe-4bab-ad6b-4e4a0a521cd4/volumes" Nov 02 15:20:45.240746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:45.240732 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" path="/var/lib/kubelet/pods/88cb0a4e-39fc-4361-bf2f-383f36dc6f13/volumes" Nov 02 15:20:46.120840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:46.120753 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:48.886145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:48.886110 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" podUID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" containerName="payload-processing" containerID="cri-o://16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593" gracePeriod=25 Nov 02 15:20:49.180682 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.180654 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:20:49.274632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.274591 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\" (UniqueName: \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\") pod \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " Nov 02 15:20:49.274825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.274789 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\") pod \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\" (UID: \"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e\") " Nov 02 15:20:49.275111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.275091 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume" pod "a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" (UID: "a7fb9676-9c8d-4be8-9b80-71ec2c6d448e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:49.275423 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.275407 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:49.276949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.276911 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8" pod "a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" (UID: "a7fb9676-9c8d-4be8-9b80-71ec2c6d448e"). InnerVolumeSpecName "kube-api-access-5t5t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:49.278030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.277789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:49.380007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.379962 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t5t8\" (UniqueName: \"kubernetes.io/projected/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e-kube-api-access-5t5t8\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:20:49.582901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.582868 4469 generic.go:356] "Generic (PLEG): container finished" podID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" containerID="16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593" exitCode=0 Nov 02 15:20:49.583099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.582924 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" Nov 02 15:20:49.583099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.582963 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" event={"ID":"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e","Type":"ContainerDied","Data":"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593"} Nov 02 15:20:49.583099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.583029 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps" event={"ID":"a7fb9676-9c8d-4be8-9b80-71ec2c6d448e","Type":"ContainerDied","Data":"6b57666b2e6b72cc6b16eeb0273714bb21bb0b3107f32c3dfc4083a59b53b53f"} Nov 02 15:20:49.583099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.583058 4469 scope.go:122] "RemoveContainer" containerID="16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593" Nov 02 15:20:49.597346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.597330 4469 scope.go:122] "RemoveContainer" containerID="16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593" Nov 02 15:20:49.597642 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:20:49.597622 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593\": container with ID starting with 16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593 not found: ID does not exist" containerID="16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593" Nov 02 15:20:49.597688 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.597649 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593"} err="failed to get container status \"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593\": rpc error: code = NotFound desc = could not find container \"16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593\": container with ID starting with 16fef4635454c89a0e54327748f04df46566da6ec6dbb78f23be8dcd4d345593 not found: ID does not exist" Nov 02 15:20:49.754068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.754038 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:20:49.812949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:49.812898 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ccdeb27-86b6749bc7-wxdps"] Nov 02 15:20:51.129893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:51.129842 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:51.240031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:51.240000 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" path="/var/lib/kubelet/pods/a7fb9676-9c8d-4be8-9b80-71ec2c6d448e/volumes" Nov 02 15:20:54.006071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:54.006020 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:58.865609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:20:58.861689 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:00.234815 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:00.234772 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:03.673494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:03.673064 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:04.491935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:04.491903 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:04.694700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:04.694652 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:05.584187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:05.584152 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:05.807315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:05.807281 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:06.187684 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:06.187650 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:21:07.839946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:07.839905 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:09.041920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:09.041876 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:21:09.061298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:09.061268 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:09.485530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:09.485490 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:09.802640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:09.802545 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:10.428095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:10.428005 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:11.154638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:11.154601 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:11.387768 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:11.387731 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:11.599018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:11.598944 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:11.900180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:11.900088 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:12.413406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:12.413369 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:12.633851 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:12.633816 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:12.855260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:12.855223 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:14.013210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.013157 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:21:14.014937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.014908 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" containerName="payload-processing" Nov 02 15:21:14.014937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.014938 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" containerName="payload-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.014965 4469 cpu_manager.go:414] "containerMap: removing container" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerName="payload-pre-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.014973 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerName="payload-pre-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015044 4469 cpu_manager.go:414] "containerMap: removing container" podUID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" containerName="payload-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015052 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" containerName="payload-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015111 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" containerName="payload-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015118 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" containerName="payload-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015137 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d2410c98-0b74-4298-bf88-07e6bcdabb94" containerName="payload-pre-processing" Nov 02 15:21:14.015175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015144 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2410c98-0b74-4298-bf88-07e6bcdabb94" containerName="payload-pre-processing" Nov 02 15:21:14.015751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015473 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="88cb0a4e-39fc-4361-bf2f-383f36dc6f13" containerName="payload-processing" Nov 02 15:21:14.015751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015503 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fb9676-9c8d-4be8-9b80-71ec2c6d448e" containerName="payload-processing" Nov 02 15:21:14.015751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015521 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2410c98-0b74-4298-bf88-07e6bcdabb94" containerName="payload-pre-processing" Nov 02 15:21:14.015751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015546 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b4d2f7-0755-4c72-8b9c-235c82b250fd" containerName="payload-processing" Nov 02 15:21:14.015751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.015564 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="1495bbff-0afe-4bab-ad6b-4e4a0a521cd4" containerName="payload-pre-processing" Nov 02 15:21:14.021008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.020982 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.091119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.088204 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-748f2e23-openshift-default-dockercfg-9p772\"" Nov 02 15:21:14.130489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.130455 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.216521 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219150 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9ns\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219195 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219242 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219340 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219360 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219431 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219464 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219489 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.219694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.219514 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322249 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322307 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322430 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322486 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322533 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322593 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322713 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9ns\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322773 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322784 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322866 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.322885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.322875 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.323095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.323073 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.323189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.323172 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.323303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.323186 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.332298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.332270 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.333025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.332958 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.387341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.387304 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.387517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.387428 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9ns\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\") pod \"e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.636683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.636596 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:14.751909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:14.751754 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:15.109933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.109900 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:21:15.110316 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:21:15.110123 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cdd77b_c55c_4506_b3c4_13365998d0fa.slice/crio-d69e719bcc74128c5e67ae320355d491dd357a880cdfc312a6102b905b46e328 WatchSource:0}: Error finding container d69e719bcc74128c5e67ae320355d491dd357a880cdfc312a6102b905b46e328: Status 404 returned error can't find the container with id d69e719bcc74128c5e67ae320355d491dd357a880cdfc312a6102b905b46e328 Nov 02 15:21:15.112885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.112851 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:21:15.112967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.112936 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:21:15.113005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.112977 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:21:15.847266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.847229 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerStarted","Data":"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6"} Nov 02 15:21:15.847469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.847287 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerStarted","Data":"d69e719bcc74128c5e67ae320355d491dd357a880cdfc312a6102b905b46e328"} Nov 02 15:21:15.977042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:15.976995 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podStartSLOduration=2.976981361 podStartE2EDuration="2.976981361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:15.974434613 +0000 UTC m=+1478.858037129" watchObservedRunningTime="2026-11-02 15:21:15.976981361 +0000 UTC m=+1478.860583879" Nov 02 15:21:16.637123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:16.637088 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:16.643052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:16.643026 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:16.855664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:16.855636 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:16.857149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:16.857125 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:21:17.123186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.123149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:17.170881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.170851 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:21:17.171187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.171150 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" podUID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" containerName="maas-api" containerID="cri-o://69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774" gracePeriod=30 Nov 02 15:21:17.478591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.478493 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:17.748321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.748251 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:17.748937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.748907 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:21:17.848661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.848629 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") pod \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " Nov 02 15:21:17.848904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.848783 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\") pod \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " Nov 02 15:21:17.848904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.848882 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\") pod \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " Nov 02 15:21:17.849068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.848908 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\" (UniqueName: \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\") pod \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\" (UID: \"000b48f0-6229-4cac-b3d3-48b4bacf52c9\") " Nov 02 15:21:17.849324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.849295 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca" pod "000b48f0-6229-4cac-b3d3-48b4bacf52c9" (UID: "000b48f0-6229-4cac-b3d3-48b4bacf52c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:17.849759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.849735 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b48f0-6229-4cac-b3d3-48b4bacf52c9-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.863974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.863943 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs" pod "000b48f0-6229-4cac-b3d3-48b4bacf52c9" (UID: "000b48f0-6229-4cac-b3d3-48b4bacf52c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:17.864135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.863971 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls" pod "000b48f0-6229-4cac-b3d3-48b4bacf52c9" (UID: "000b48f0-6229-4cac-b3d3-48b4bacf52c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:17.864211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.864045 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9" pod "000b48f0-6229-4cac-b3d3-48b4bacf52c9" (UID: "000b48f0-6229-4cac-b3d3-48b4bacf52c9"). InnerVolumeSpecName "kube-api-access-p25v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:17.878128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.877490 4469 generic.go:356] "Generic (PLEG): container finished" podID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" containerID="69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774" exitCode=0 Nov 02 15:21:17.888097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.881537 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" Nov 02 15:21:17.888097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.887075 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" event={"ID":"000b48f0-6229-4cac-b3d3-48b4bacf52c9","Type":"ContainerDied","Data":"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774"} Nov 02 15:21:17.888097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.887151 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh" event={"ID":"000b48f0-6229-4cac-b3d3-48b4bacf52c9","Type":"ContainerDied","Data":"e7a228fc5dcbbed6c3346a8d8e951cf06c4969d2c140e6063333c202ea61feeb"} Nov 02 15:21:17.888097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.887185 4469 scope.go:122] "RemoveContainer" containerID="69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774" Nov 02 15:21:17.926092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.926067 4469 scope.go:122] "RemoveContainer" containerID="69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774" Nov 02 15:21:17.926471 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:17.926445 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774\": container with ID starting with 69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774 not found: ID does not exist" containerID="69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774" Nov 02 15:21:17.926598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.926477 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774"} err="failed to get container status \"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774\": rpc error: code = NotFound desc = could not find container \"69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774\": container with ID starting with 69368b8a86d84beae56d2eaa2152c3cb66c17b48319ea4be6b53f6fc6814f774 not found: ID does not exist" Nov 02 15:21:17.951683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.951652 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.951683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.951685 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25v9\" (UniqueName: \"kubernetes.io/projected/000b48f0-6229-4cac-b3d3-48b4bacf52c9-kube-api-access-p25v9\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.951986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:17.951699 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000b48f0-6229-4cac-b3d3-48b4bacf52c9-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:18.060589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:18.060457 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:21:18.065401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:18.065326 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ccdeb27-64d4f887c5-pkxmh"] Nov 02 15:21:18.243940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:18.243903 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:18.558823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:18.558242 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:19.242148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:19.242110 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" path="/var/lib/kubelet/pods/000b48f0-6229-4cac-b3d3-48b4bacf52c9/volumes" Nov 02 15:21:19.717709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:19.716020 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:20.316012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:20.315953 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:20.670917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:20.668768 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:20.925942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:20.923978 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:20.999774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:20.999744 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:21:21.004529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.004463 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" containerID="cri-o://7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea" gracePeriod=30 Nov 02 15:21:21.040873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.018147 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:21:21.040873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.020142 4469 cpu_manager.go:414] "containerMap: removing container" podUID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" containerName="maas-api" Nov 02 15:21:21.040873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.020165 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" containerName="maas-api" Nov 02 15:21:21.040873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.020487 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="000b48f0-6229-4cac-b3d3-48b4bacf52c9" containerName="maas-api" Nov 02 15:21:21.060312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.060277 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.108345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.108308 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:21:21.118856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.115290 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5cbda1cc-dockercfg-hmvw8\"" Nov 02 15:21:21.162991 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.162953 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:21:21.173765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.173419 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:21.174562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.174539 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.210904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.210872 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5cbda1cc\"" Nov 02 15:21:21.218458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.218434 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:21:21.243609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.243521 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.244774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.244743 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqddt\" (UniqueName: \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.346679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.346641 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm92\" (UniqueName: \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.347224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.346701 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqddt\" (UniqueName: \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.347224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.346953 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.347224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.347064 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.348254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.348224 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.415507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.415473 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqddt\" (UniqueName: \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\") pod \"payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.449518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.449422 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.449677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.449581 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm92\" (UniqueName: \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.450392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.450365 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.512661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.512629 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm92\" (UniqueName: \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\") pod \"payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:21.686183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.686147 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:21.794333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:21.793152 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:22.014065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.014013 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:22.076977 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:21:22.076945 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd879dd65_0253_490c_a574_c916c01b576e.slice/crio-c0d9b7dafad39853175ef4225b552e8566120b6096e54bf329ed7394ed2638e0 WatchSource:0}: Error finding container c0d9b7dafad39853175ef4225b552e8566120b6096e54bf329ed7394ed2638e0: Status 404 returned error can't find the container with id c0d9b7dafad39853175ef4225b552e8566120b6096e54bf329ed7394ed2638e0 Nov 02 15:21:22.077284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.077260 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:21:22.150648 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.150620 4469 patch_prober.go:29] interesting pod/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:21:22.150757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.150672 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:21:22.193930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.193902 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:21:22.194294 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:21:22.194271 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cda9da2_1b7f_4174_a733_22dcf28b0231.slice/crio-312066e7dcccdecea128caf8c023d781500d69ec2615e2241236a20d871c7f69 WatchSource:0}: Error finding container 312066e7dcccdecea128caf8c023d781500d69ec2615e2241236a20d871c7f69: Status 404 returned error can't find the container with id 312066e7dcccdecea128caf8c023d781500d69ec2615e2241236a20d871c7f69 Nov 02 15:21:22.966170 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.966137 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" event={"ID":"d879dd65-0253-490c-a574-c916c01b576e","Type":"ContainerStarted","Data":"fb8ddfc31cf3f391022178358fda8af19bff6a2912710430ab8371bbc5060759"} Nov 02 15:21:22.966647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.966200 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" event={"ID":"d879dd65-0253-490c-a574-c916c01b576e","Type":"ContainerStarted","Data":"c0d9b7dafad39853175ef4225b552e8566120b6096e54bf329ed7394ed2638e0"} Nov 02 15:21:22.966647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.966348 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:22.968017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.967991 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" event={"ID":"2cda9da2-1b7f-4174-a733-22dcf28b0231","Type":"ContainerStarted","Data":"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853"} Nov 02 15:21:22.968134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.968038 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" event={"ID":"2cda9da2-1b7f-4174-a733-22dcf28b0231","Type":"ContainerStarted","Data":"312066e7dcccdecea128caf8c023d781500d69ec2615e2241236a20d871c7f69"} Nov 02 15:21:22.968134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:22.968068 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:23.085652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:23.085584 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" podStartSLOduration=3.085567688 podStartE2EDuration="3.085567688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:23.083459976 +0000 UTC m=+1485.967062496" watchObservedRunningTime="2026-11-02 15:21:23.085567688 +0000 UTC m=+1485.969170208" Nov 02 15:21:23.418028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:23.417945 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:25.362363 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:25.362311 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:25.947840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:25.947776 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:26.268013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.240517 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:26.323017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.322949 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" podStartSLOduration=5.322929204 podStartE2EDuration="5.322929204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:23.217376875 +0000 UTC m=+1486.100979395" watchObservedRunningTime="2026-11-02 15:21:26.322929204 +0000 UTC m=+1489.206531724" Nov 02 15:21:26.323346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.323323 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:21:26.333025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.332986 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.405017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.402520 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:21:26.503698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.477950 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvn6\" (UniqueName: \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.503698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.478117 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.503698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.499733 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:21:26.504746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.504696 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-748f2e23\"" Nov 02 15:21:26.504952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.504842 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-748f2e23-dockercfg-z959c\"" Nov 02 15:21:26.539796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.539748 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:21:26.541304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.541264 4469 cpu_manager.go:414] "containerMap: removing container" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.541502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.541488 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.541734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.541720 4469 cpu_manager.go:414] "containerMap: removing container" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.541849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.541837 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.542237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.542221 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.542382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.542370 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerName="istio-proxy" Nov 02 15:21:26.547153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.547134 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.571606 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.571547 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:21:26.579765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.579733 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvn6\" (UniqueName: \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.580013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.579898 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.580858 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.580834 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.644882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.644840 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvn6\" (UniqueName: \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\") pod \"payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.662889 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.662863 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:26.682674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.682633 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.682771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.682736 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.682881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.682831 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.682958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.682937 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683023 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683107 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683195 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683237 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683273 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\") pod \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\" (UID: \"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7\") " Nov 02 15:21:26.683666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683642 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxw95\" (UniqueName: \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.683945 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.683925 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.688410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.688383 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:26.689655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.689462 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:26.689865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.689814 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:26.690058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.689993 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:26.690311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.690290 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:26.693085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.693051 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "kube-api-access-7splh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:26.695153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.695115 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:26.697330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.697277 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:21:26.700975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.698923 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket" pod "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" (UID: "f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:26.787392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787303 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxw95\" (UniqueName: \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.787765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787736 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.787909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787858 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.787909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787878 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.787909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787893 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787907 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7splh\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-kube-api-access-7splh\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787922 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787935 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787948 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787961 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.787976 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:21:26.788749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.788724 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.866203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.866114 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxw95\" (UniqueName: \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\") pod \"payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:26.918994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:26.900162 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:27.138609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.137538 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:21:27.156690 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.156663 4469 generic.go:356] "Generic (PLEG): container finished" podID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" containerID="7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea" exitCode=0 Nov 02 15:21:27.157062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.156883 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerDied","Data":"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea"} Nov 02 15:21:27.157234 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.157217 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" event={"ID":"f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7","Type":"ContainerDied","Data":"c9d64f45a0febd375c571880d78a85376aa717d05279f42ec9a90cd39aba089c"} Nov 02 15:21:27.157354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.157341 4469 scope.go:122] "RemoveContainer" containerID="7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea" Nov 02 15:21:27.157534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.156930 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb" Nov 02 15:21:27.176320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.176293 4469 scope.go:122] "RemoveContainer" containerID="118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de" Nov 02 15:21:27.189312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.189291 4469 scope.go:122] "RemoveContainer" containerID="7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea" Nov 02 15:21:27.189587 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:27.189570 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea\": container with ID starting with 7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea not found: ID does not exist" containerID="7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea" Nov 02 15:21:27.189631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.189594 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea"} err="failed to get container status \"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea\": rpc error: code = NotFound desc = could not find container \"7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea\": container with ID starting with 7365af3133dd96bf569b1fffdceaf0cdf0f0cc9d5449aa2e79edd3f8fa8e17ea not found: ID does not exist" Nov 02 15:21:27.189631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.189612 4469 scope.go:122] "RemoveContainer" containerID="118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de" Nov 02 15:21:27.189830 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:27.189796 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de\": container with ID starting with 118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de not found: ID does not exist" containerID="118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de" Nov 02 15:21:27.189830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.189829 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de"} err="failed to get container status \"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de\": rpc error: code = NotFound desc = could not find container \"118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de\": container with ID starting with 118be8e8cf39fc1dc7379741ac1b8ccc3a27eb3a395ffa7750c57ffd9641f5de not found: ID does not exist" Nov 02 15:21:27.324993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.324903 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:21:27.328647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.328619 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5ccdeb27-openshift-default-6b98454f65-tdjwb"] Nov 02 15:21:27.387740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:27.387714 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:21:27.387892 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:21:27.387867 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6253bf9_f6a3_41ff_a9a1_6ebed1f5d87e.slice/crio-5c6f245941edb88ed5810018ec12049038aa3e68cb066ea897dc0d71b198607b WatchSource:0}: Error finding container 5c6f245941edb88ed5810018ec12049038aa3e68cb066ea897dc0d71b198607b: Status 404 returned error can't find the container with id 5c6f245941edb88ed5810018ec12049038aa3e68cb066ea897dc0d71b198607b Nov 02 15:21:28.166519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.166477 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" event={"ID":"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e","Type":"ContainerStarted","Data":"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817"} Nov 02 15:21:28.166998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.166577 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" event={"ID":"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e","Type":"ContainerStarted","Data":"5c6f245941edb88ed5810018ec12049038aa3e68cb066ea897dc0d71b198607b"} Nov 02 15:21:28.166998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.166730 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:28.168295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.168264 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" event={"ID":"0d470abe-ca6d-45fe-9384-f232bca0888a","Type":"ContainerStarted","Data":"40fd917433b01c1c34bf2390cce8a0d478220ccfb6c3fe43bc84fb5ca6257e33"} Nov 02 15:21:28.168393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.168319 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" event={"ID":"0d470abe-ca6d-45fe-9384-f232bca0888a","Type":"ContainerStarted","Data":"77d045f05426aaa6cafd0c7c5dd2504dcbbd870934ff5e15eaf8db96c6466ed2"} Nov 02 15:21:28.168393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.168352 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:28.288759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.288703 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" podStartSLOduration=2.288688063 podStartE2EDuration="2.288688063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:28.284796201 +0000 UTC m=+1491.168398720" watchObservedRunningTime="2026-11-02 15:21:28.288688063 +0000 UTC m=+1491.172290626" Nov 02 15:21:28.414956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:28.414003 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" podStartSLOduration=2.413980354 podStartE2EDuration="2.413980354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:28.412336186 +0000 UTC m=+1491.295938706" watchObservedRunningTime="2026-11-02 15:21:28.413980354 +0000 UTC m=+1491.297582871" Nov 02 15:21:29.239546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:29.239516 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7" path="/var/lib/kubelet/pods/f2c079aa-3df8-4b44-93bb-bb7e3a7fc4d7/volumes" Nov 02 15:21:30.266218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:30.266175 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:33.978776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:33.978745 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:21:33.979217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:33.979198 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:21:35.169228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:35.169190 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:36.187616 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:36.187581 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:21:39.181585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:39.181538 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:21:39.182063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:39.182033 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:21:40.284524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:40.284478 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:41.550306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:41.550265 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:45.180333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:45.180284 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:50.069879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:50.069782 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:51.474508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:51.474468 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:55.139060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:55.138988 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:58.635744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:58.635672 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:59.640747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.640714 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:21:59.648125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.648097 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.705037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.705003 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:21:59.731652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.731622 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.731790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.731727 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svg8f\" (UniqueName: \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.731790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.731752 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.731962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.731945 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.750147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.750119 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5cbda1cc-serving-cert\"" Nov 02 15:21:59.833638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.833602 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.833831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.833670 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svg8f\" (UniqueName: \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.833831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.833697 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.833917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.833885 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.833952 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:59.833906 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-serving-cert: secret "maas-api-e2e-ait-5cbda1cc-serving-cert" not found Nov 02 15:21:59.834014 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:21:59.834000 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls podName:6fba9515-49a6-441f-82c1-f6b114b274e9 nodeName:}" failed. No retries permitted until 2026-11-02 15:22:00.333976497 +0000 UTC m=+1523.217579008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls") pod "maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" (UID: "6fba9515-49a6-441f-82c1-f6b114b274e9") : secret "maas-api-e2e-ait-5cbda1cc-serving-cert" not found Nov 02 15:21:59.834355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.834333 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.836391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.836367 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:21:59.909077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.906203 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:59.911873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:21:59.911843 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svg8f\" (UniqueName: \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:00.150017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.149978 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:00.347170 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.347135 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:00.349393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.349372 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") pod \"maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:00.564537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.564496 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:00.950516 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:00.950491 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fba9515_49a6_441f_82c1_f6b114b274e9.slice/crio-bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4 WatchSource:0}: Error finding container bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4: Status 404 returned error can't find the container with id bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4 Nov 02 15:22:00.952263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.952239 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:22:00.952753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:00.952731 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:22:01.490336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:01.490292 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" event={"ID":"6fba9515-49a6-441f-82c1-f6b114b274e9","Type":"ContainerStarted","Data":"bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4"} Nov 02 15:22:02.501138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:02.501098 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" event={"ID":"6fba9515-49a6-441f-82c1-f6b114b274e9","Type":"ContainerStarted","Data":"340741cdb90e0c3eafaa7d527245e88948f0752ed21e030b60e64bc593aad3bf"} Nov 02 15:22:02.501608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:02.501205 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:02.627569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:02.627518 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" podStartSLOduration=2.7237914439999997 podStartE2EDuration="3.627502377s" totalImagesPullingTime="903.710933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:02.625016339 +0000 UTC m=+1525.508618881" watchObservedRunningTime="2026-11-02 15:22:02.627502377 +0000 UTC m=+1525.511104909" Nov 02 15:22:04.633376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.633343 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:22:04.637824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.637782 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.718020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.717983 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.718198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.718026 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.718275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.718254 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.718352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.718296 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmxz7\" (UniqueName: \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.742074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.742040 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-748f2e23-serving-cert\"" Nov 02 15:22:04.746504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.746477 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:22:04.820061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.820022 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.820061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.820062 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.820307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.820196 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.820307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.820224 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmxz7\" (UniqueName: \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.820307 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:22:04.820191 4469 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-serving-cert: secret "maas-api-e2e-ait-748f2e23-serving-cert" not found Nov 02 15:22:04.820470 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:22:04.820343 4469 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls podName:4ac1fc12-6fd3-4657-a91d-7ed11109e952 nodeName:}" failed. No retries permitted until 2026-11-02 15:22:05.320319399 +0000 UTC m=+1528.203921912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls") pod "maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" (UID: "4ac1fc12-6fd3-4657-a91d-7ed11109e952") : secret "maas-api-e2e-ait-748f2e23-serving-cert" not found Nov 02 15:22:04.820939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.820915 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.822794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.822772 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.886712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.886648 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmxz7\" (UniqueName: \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:04.901677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:04.901645 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:05.108131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.108091 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:05.328410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.328378 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:05.330710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.330686 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") pod \"maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:05.552538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.552503 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:05.659703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.659665 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:05.945955 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:05.945927 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac1fc12_6fd3_4657_a91d_7ed11109e952.slice/crio-851ad8446c5cb7ebc9eaa5be914d0a776284bbb884a635d1b587a633bf049a92 WatchSource:0}: Error finding container 851ad8446c5cb7ebc9eaa5be914d0a776284bbb884a635d1b587a633bf049a92: Status 404 returned error can't find the container with id 851ad8446c5cb7ebc9eaa5be914d0a776284bbb884a635d1b587a633bf049a92 Nov 02 15:22:05.978445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:05.978420 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:22:06.199336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.197296 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:22:06.217674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.217647 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.270439 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:22:06.269705 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:22:06.280459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.280354 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:22:06.321593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.321562 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-be5d503e-openshift-default-dockercfg-zvvgw\"" Nov 02 15:22:06.341846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.341205 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.353985 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354053 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354099 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354170 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354293 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354343 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354387 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354459 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.359708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.354504 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lw7p\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456231 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456312 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456358 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456432 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456478 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lw7p\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456786 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.456950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456917 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.457008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.456965 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.457065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.457047 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.457856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.457220 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.457856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.457439 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.457856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.457660 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.458247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.458073 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.458247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.458078 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.461184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.461154 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.461366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.461347 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.524917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.524862 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lw7p\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.525106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.524938 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\") pod \"e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.535771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.535738 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" event={"ID":"4ac1fc12-6fd3-4657-a91d-7ed11109e952","Type":"ContainerStarted","Data":"851ad8446c5cb7ebc9eaa5be914d0a776284bbb884a635d1b587a633bf049a92"} Nov 02 15:22:06.591871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.591840 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:06.646518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:06.645102 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:07.065602 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:07.065572 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8c4622_0815_477e_abf4_bfac91d7c620.slice/crio-4420bc449bb99c6ecda19cfe132c80ac8c51a010a911a527a2c47868feda6d80 WatchSource:0}: Error finding container 4420bc449bb99c6ecda19cfe132c80ac8c51a010a911a527a2c47868feda6d80: Status 404 returned error can't find the container with id 4420bc449bb99c6ecda19cfe132c80ac8c51a010a911a527a2c47868feda6d80 Nov 02 15:22:07.068622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.068577 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:07.068738 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.068697 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:07.068843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.068737 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:07.068988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.068961 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:22:07.555612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.555531 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerStarted","Data":"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b"} Nov 02 15:22:07.555819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.555654 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerStarted","Data":"4420bc449bb99c6ecda19cfe132c80ac8c51a010a911a527a2c47868feda6d80"} Nov 02 15:22:07.557998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.557968 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" event={"ID":"4ac1fc12-6fd3-4657-a91d-7ed11109e952","Type":"ContainerStarted","Data":"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705"} Nov 02 15:22:07.558118 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.558047 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:07.592422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.592402 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:07.677538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.677479 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podStartSLOduration=1.677462419 podStartE2EDuration="1.677462419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:07.677247658 +0000 UTC m=+1530.560850190" watchObservedRunningTime="2026-11-02 15:22:07.677462419 +0000 UTC m=+1530.561064940" Nov 02 15:22:07.804010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:07.803946 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" podStartSLOduration=2.965446967 podStartE2EDuration="3.803926509s" totalImagesPullingTime="838.479542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:07.799963681 +0000 UTC m=+1530.683566201" watchObservedRunningTime="2026-11-02 15:22:07.803926509 +0000 UTC m=+1530.687529029" Nov 02 15:22:08.512989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:08.512958 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:22:08.598380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:08.598349 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:09.577235 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:09.577201 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:09.578230 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:09.578210 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:22:10.007916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:10.007873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:10.360876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:10.360117 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:10.692980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:10.692879 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:10.913473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:10.913437 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:11.126760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.126729 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:11.771588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.771549 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:22:11.777357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.777335 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.796602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.796565 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.796726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.796649 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.796726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.796685 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.796835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.796763 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c52t\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.796937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.796921 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.797085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.797066 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.797124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.797106 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.797157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.797127 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.797248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.797234 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.830147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.830090 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-895ac8-openshift-default-dockercfg-zhsm9\"" Nov 02 15:22:11.885829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.884714 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900146 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900215 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900303 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c52t\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900598 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900776 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900831 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900842 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.900930 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.901008 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.901158 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.901516 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.901605 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.901771 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.902140 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.905293 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.912952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.912209 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:11.942855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:11.937877 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:12.005429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.005397 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c52t\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:12.005614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.005498 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\") pod \"e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:12.092752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.092641 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:12.154670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.154136 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:12.564879 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:12.564848 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbc2baa_54da_4420_967e_88fab3fe2981.slice/crio-fb1b0090be2b8a8b8eea6e97c9ecb7621cae82fd95405b0451ae250934b38246 WatchSource:0}: Error finding container fb1b0090be2b8a8b8eea6e97c9ecb7621cae82fd95405b0451ae250934b38246: Status 404 returned error can't find the container with id fb1b0090be2b8a8b8eea6e97c9ecb7621cae82fd95405b0451ae250934b38246 Nov 02 15:22:12.565988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.565953 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:22:12.567963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.567922 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:12.568077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.568011 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:12.568077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.568055 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:22:12.602727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:12.602704 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerStarted","Data":"fb1b0090be2b8a8b8eea6e97c9ecb7621cae82fd95405b0451ae250934b38246"} Nov 02 15:22:13.574448 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:13.574415 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:22:13.611657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:13.611621 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerStarted","Data":"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc"} Nov 02 15:22:13.844728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:13.844619 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podStartSLOduration=2.844603487 podStartE2EDuration="2.844603487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:13.841420467 +0000 UTC m=+1536.725022988" watchObservedRunningTime="2026-11-02 15:22:13.844603487 +0000 UTC m=+1536.728206005" Nov 02 15:22:14.110296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.109910 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:14.112241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.112141 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:14.203821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.203704 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:14.325875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.325835 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:14.353076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.353045 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:22:14.358117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.358087 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.410459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.410431 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-be5d503e\"" Nov 02 15:22:14.453825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.453599 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:22:14.479655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.479612 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.479870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.479728 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4c8l\" (UniqueName: \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.592413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.592375 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4c8l\" (UniqueName: \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.593367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.593329 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.594336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.594310 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.632879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.632736 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:14.640123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.640088 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:22:14.685409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.685374 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4c8l\" (UniqueName: \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\") pod \"payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.685593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.685577 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:22:14.692361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.692333 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:14.693584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.693561 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:14.694652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.694531 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:22:14.745842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.745411 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-be5d503e-dockercfg-9dxll\"" Nov 02 15:22:14.801464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.799021 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:14.801464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.799363 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlpt\" (UniqueName: \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:14.926247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.924134 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlpt\" (UniqueName: \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:14.926247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.924358 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:14.926247 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.925491 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:15.002705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:14.994467 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlpt\" (UniqueName: \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\") pod \"payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:15.111381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.111353 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:15.301882 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:15.301848 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638cbdb8_5061_4918_8324_ad3f8ebfd0dc.slice/crio-9c13936ff69e21e2ec54359e25f14d9c9eca314f15ab2e6d8bdd018242032d22 WatchSource:0}: Error finding container 9c13936ff69e21e2ec54359e25f14d9c9eca314f15ab2e6d8bdd018242032d22: Status 404 returned error can't find the container with id 9c13936ff69e21e2ec54359e25f14d9c9eca314f15ab2e6d8bdd018242032d22 Nov 02 15:22:15.304322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.304298 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:22:15.490837 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:15.490779 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8080f2a4_75c1_4e88_acd2_c0ba8aa05717.slice/crio-9abae77564fdf91d43e24a83f7a39cf55f4fd7e696c6d5874a90de60a5ff8055 WatchSource:0}: Error finding container 9abae77564fdf91d43e24a83f7a39cf55f4fd7e696c6d5874a90de60a5ff8055: Status 404 returned error can't find the container with id 9abae77564fdf91d43e24a83f7a39cf55f4fd7e696c6d5874a90de60a5ff8055 Nov 02 15:22:15.490935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.490858 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:22:15.599678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.599643 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:22:15.641182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.641143 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" event={"ID":"8080f2a4-75c1-4e88-acd2-c0ba8aa05717","Type":"ContainerStarted","Data":"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4"} Nov 02 15:22:15.641343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.641204 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" event={"ID":"8080f2a4-75c1-4e88-acd2-c0ba8aa05717","Type":"ContainerStarted","Data":"9abae77564fdf91d43e24a83f7a39cf55f4fd7e696c6d5874a90de60a5ff8055"} Nov 02 15:22:15.641343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.641230 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:15.643052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.643019 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" event={"ID":"638cbdb8-5061-4918-8324-ad3f8ebfd0dc","Type":"ContainerStarted","Data":"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97"} Nov 02 15:22:15.643181 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.643078 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" event={"ID":"638cbdb8-5061-4918-8324-ad3f8ebfd0dc","Type":"ContainerStarted","Data":"9c13936ff69e21e2ec54359e25f14d9c9eca314f15ab2e6d8bdd018242032d22"} Nov 02 15:22:15.643250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.643237 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:15.759752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.759691 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" podStartSLOduration=1.759675176 podStartE2EDuration="1.759675176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:15.757289423 +0000 UTC m=+1538.640891943" watchObservedRunningTime="2026-11-02 15:22:15.759675176 +0000 UTC m=+1538.643277743" Nov 02 15:22:15.899360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.899234 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" podStartSLOduration=1.899202496 podStartE2EDuration="1.899202496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:15.886794594 +0000 UTC m=+1538.770397114" watchObservedRunningTime="2026-11-02 15:22:15.899202496 +0000 UTC m=+1538.782805017" Nov 02 15:22:15.906848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:15.906787 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:16.865183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:16.865116 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:17.116600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:17.116508 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:17.416721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:17.416641 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.181270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.181226 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.405528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.405492 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.741224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.725328 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:22:20.807822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.796507 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:22:20.808339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.808311 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:20.886821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.885234 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-895ac8\"" Nov 02 15:22:20.897699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.887392 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldv86\" (UniqueName: \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:20.897699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.887707 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:20.925653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.925619 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.934024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.934001 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-895ac8-dockercfg-qrw55\"" Nov 02 15:22:20.990848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.989600 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldv86\" (UniqueName: \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:20.990848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.990134 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:20.991420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:20.991357 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:21.109825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:21.109669 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldv86\" (UniqueName: \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\") pod \"payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:21.163860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:21.162527 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:21.650468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:21.649235 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:22:21.678891 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:22:21.674102 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad04e26_7046_41d9_8f11_b9e0a6d0cc01.slice/crio-f9a3c8ee4687af0354d5d4489fee68db7029162943d3fdbeb5254267c0b0b745 WatchSource:0}: Error finding container f9a3c8ee4687af0354d5d4489fee68db7029162943d3fdbeb5254267c0b0b745: Status 404 returned error can't find the container with id f9a3c8ee4687af0354d5d4489fee68db7029162943d3fdbeb5254267c0b0b745 Nov 02 15:22:21.734984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:21.734675 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" event={"ID":"fad04e26-7046-41d9-8f11-b9e0a6d0cc01","Type":"ContainerStarted","Data":"f9a3c8ee4687af0354d5d4489fee68db7029162943d3fdbeb5254267c0b0b745"} Nov 02 15:22:21.930772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:21.930681 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:22.177313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:22.177281 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:22.745149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:22.745103 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" event={"ID":"fad04e26-7046-41d9-8f11-b9e0a6d0cc01","Type":"ContainerStarted","Data":"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93"} Nov 02 15:22:22.745613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:22.745174 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:22.866821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:22.865028 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" podStartSLOduration=2.8650081480000003 podStartE2EDuration="2.865008148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:22.860366739 +0000 UTC m=+1545.743969255" watchObservedRunningTime="2026-11-02 15:22:22.865008148 +0000 UTC m=+1545.748610667" Nov 02 15:22:26.317043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:26.317005 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:26.574112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:26.574029 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:26.654537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:26.654497 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:22:26.655227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:26.655199 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:22:27.322453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:27.322419 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:28.277172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:28.277124 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:29.561465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:29.561428 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:29.813171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:29.813083 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:32.363791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:32.363760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:33.756523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:33.756491 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:22:36.188907 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:22:36.188864 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:22:37.131850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:37.131771 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:41.902677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:41.902639 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:42.104640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:42.104597 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:46.664419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:46.664379 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:46.908291 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:46.908255 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:51.900649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:51.897076 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:52.105847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:52.105789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:52.177618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:52.177529 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:22:52.177846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:52.177822 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" containerID="cri-o://d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c" gracePeriod=30 Nov 02 15:22:57.259459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.259418 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:57.569766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.569500 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:22:57.580880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.580855 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.580887 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.580931 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581132 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:57.581215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581202 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581253 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581274 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:57.581338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581300 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581328 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581344 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581433 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\") pod \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\" (UID: \"22984f6a-27d0-4558-bbce-8a4ac8f858f2\") " Nov 02 15:22:57.581521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581503 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:57.581666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581652 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:57.581984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.581947 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:57.582478 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.582454 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.582551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.582484 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.582551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.582500 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.582551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.582513 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.582551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.582526 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.583867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.583641 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "kube-api-access-p5glv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:57.583867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.583711 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:57.585109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.585088 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:22:57.585311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.585288 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy" pod "22984f6a-27d0-4558-bbce-8a4ac8f858f2" (UID: "22984f6a-27d0-4558-bbce-8a4ac8f858f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:57.684242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.684206 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.684242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.684234 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.684242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.684244 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22984f6a-27d0-4558-bbce-8a4ac8f858f2-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.684479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:57.684256 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5glv\" (UniqueName: \"kubernetes.io/projected/22984f6a-27d0-4558-bbce-8a4ac8f858f2-kube-api-access-p5glv\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:22:58.066934 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.066903 4469 generic.go:356] "Generic (PLEG): container finished" podID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerID="d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c" exitCode=0 Nov 02 15:22:58.067105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.066973 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" Nov 02 15:22:58.067105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.066990 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" event={"ID":"22984f6a-27d0-4558-bbce-8a4ac8f858f2","Type":"ContainerDied","Data":"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c"} Nov 02 15:22:58.067105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.067045 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj" event={"ID":"22984f6a-27d0-4558-bbce-8a4ac8f858f2","Type":"ContainerDied","Data":"9201d7353dd4e2e75abc60e0d798627474c9569691dcd157e658bd1767b90958"} Nov 02 15:22:58.067105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.067072 4469 scope.go:122] "RemoveContainer" containerID="d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c" Nov 02 15:22:58.081212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.081197 4469 scope.go:122] "RemoveContainer" containerID="d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c" Nov 02 15:22:58.081468 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:22:58.081448 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c\": container with ID starting with d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c not found: ID does not exist" containerID="d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c" Nov 02 15:22:58.081534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.081478 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c"} err="failed to get container status \"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c\": rpc error: code = NotFound desc = could not find container \"d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c\": container with ID starting with d0ec091b034fc5582cb6b060a168e7c8582952c8a186b9c23bb82b53525a2c9c not found: ID does not exist" Nov 02 15:22:58.242754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.242719 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:22:58.246341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:58.246303 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-bnsmj"] Nov 02 15:22:59.243176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:22:59.243144 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" path="/var/lib/kubelet/pods/22984f6a-27d0-4558-bbce-8a4ac8f858f2/volumes" Nov 02 15:23:01.865883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:01.865843 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:01.866418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:01.866224 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:02.100443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.100365 4469 generic.go:356] "Generic (PLEG): container finished" podID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerID="635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6" exitCode=0 Nov 02 15:23:02.100591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.100446 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerDied","Data":"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6"} Nov 02 15:23:02.101036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.101015 4469 scope.go:122] "RemoveContainer" containerID="635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6" Nov 02 15:23:02.102754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.102724 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:02.102893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.102794 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:02.102893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.102872 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:02.556586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.556376 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:02.774130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:02.774091 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.110242 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:03.110209 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerStarted","Data":"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb"} Nov 02 15:23:03.640305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:03.640271 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:23:03.640671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:03.640646 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:03.640864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:03.640838 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:04.638825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:04.637317 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:04.638825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:04.637384 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:05.583606 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:05.583575 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:05.646824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:05.646359 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:05.646824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:05.646438 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:06.190119 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:23:06.190069 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:23:06.644823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:06.644174 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:06.644823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:06.644251 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:07.637208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:07.637168 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:07.637793 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:07.637757 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:08.396451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:08.396418 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:08.638019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:08.637975 4469 patch_prober.go:29] interesting pod/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 15:23:08.638529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:08.638058 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 15:23:09.642281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:09.642249 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:23:09.642730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:09.642708 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:23:09.643439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:09.643419 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:23:12.837842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:12.837788 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:23:12.954360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:12.954277 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:23:13.282742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:13.282709 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:13.495691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:13.495653 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:13.721762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:13.721730 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:13.931085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:13.931047 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:14.144859 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:14.144765 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:17.838160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:17.838108 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" podUID="2cda9da2-1b7f-4174-a733-22dcf28b0231" containerName="payload-processing" containerID="cri-o://68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853" gracePeriod=25 Nov 02 15:23:17.951465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:17.951430 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:23:17.955002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:17.954968 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" podUID="d879dd65-0253-490c-a574-c916c01b576e" containerName="payload-pre-processing" containerID="cri-o://fb8ddfc31cf3f391022178358fda8af19bff6a2912710430ab8371bbc5060759" gracePeriod=25 Nov 02 15:23:18.060656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.060230 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:23:18.145754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.145724 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:23:18.303534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.303499 4469 generic.go:356] "Generic (PLEG): container finished" podID="d879dd65-0253-490c-a574-c916c01b576e" containerID="fb8ddfc31cf3f391022178358fda8af19bff6a2912710430ab8371bbc5060759" exitCode=0 Nov 02 15:23:18.303722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.303585 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" event={"ID":"d879dd65-0253-490c-a574-c916c01b576e","Type":"ContainerDied","Data":"fb8ddfc31cf3f391022178358fda8af19bff6a2912710430ab8371bbc5060759"} Nov 02 15:23:18.305271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.305246 4469 generic.go:356] "Generic (PLEG): container finished" podID="2cda9da2-1b7f-4174-a733-22dcf28b0231" containerID="68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853" exitCode=0 Nov 02 15:23:18.305397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.305277 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" event={"ID":"2cda9da2-1b7f-4174-a733-22dcf28b0231","Type":"ContainerDied","Data":"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853"} Nov 02 15:23:18.305397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.305301 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" Nov 02 15:23:18.305397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.305327 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7" event={"ID":"2cda9da2-1b7f-4174-a733-22dcf28b0231","Type":"ContainerDied","Data":"312066e7dcccdecea128caf8c023d781500d69ec2615e2241236a20d871c7f69"} Nov 02 15:23:18.305397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.305353 4469 scope.go:122] "RemoveContainer" containerID="68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853" Nov 02 15:23:18.307677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.307635 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\") pod \"2cda9da2-1b7f-4174-a733-22dcf28b0231\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " Nov 02 15:23:18.308269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.308242 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume" pod "2cda9da2-1b7f-4174-a733-22dcf28b0231" (UID: "2cda9da2-1b7f-4174-a733-22dcf28b0231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:18.308739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.308720 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\" (UniqueName: \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\") pod \"2cda9da2-1b7f-4174-a733-22dcf28b0231\" (UID: \"2cda9da2-1b7f-4174-a733-22dcf28b0231\") " Nov 02 15:23:18.309463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.309441 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cda9da2-1b7f-4174-a733-22dcf28b0231-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:18.314377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.314348 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92" pod "2cda9da2-1b7f-4174-a733-22dcf28b0231" (UID: "2cda9da2-1b7f-4174-a733-22dcf28b0231"). InnerVolumeSpecName "kube-api-access-7jm92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:18.337679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.333889 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:23:18.339059 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.339035 4469 scope.go:122] "RemoveContainer" containerID="68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853" Nov 02 15:23:18.344983 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:23:18.344947 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853\": container with ID starting with 68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853 not found: ID does not exist" containerID="68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853" Nov 02 15:23:18.345115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.344990 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853"} err="failed to get container status \"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853\": rpc error: code = NotFound desc = could not find container \"68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853\": container with ID starting with 68bb9fde0987b0cff9b941b8167fca6b11eeaedd21bdf26891b0e2157147a853 not found: ID does not exist" Nov 02 15:23:18.370872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.364990 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:18.429832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.424511 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jm92\" (UniqueName: \"kubernetes.io/projected/2cda9da2-1b7f-4174-a733-22dcf28b0231-kube-api-access-7jm92\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:18.525165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.525132 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\") pod \"d879dd65-0253-490c-a574-c916c01b576e\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " Nov 02 15:23:18.525421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.525401 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\" (UniqueName: \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\") pod \"d879dd65-0253-490c-a574-c916c01b576e\" (UID: \"d879dd65-0253-490c-a574-c916c01b576e\") " Nov 02 15:23:18.525572 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.525549 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume" pod "d879dd65-0253-490c-a574-c916c01b576e" (UID: "d879dd65-0253-490c-a574-c916c01b576e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:18.526026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.526008 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d879dd65-0253-490c-a574-c916c01b576e-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:18.528240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.528213 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt" pod "d879dd65-0253-490c-a574-c916c01b576e" (UID: "d879dd65-0253-490c-a574-c916c01b576e"). InnerVolumeSpecName "kube-api-access-wqddt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:18.572310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.572284 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:18.627730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.627645 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqddt\" (UniqueName: \"kubernetes.io/projected/d879dd65-0253-490c-a574-c916c01b576e-kube-api-access-wqddt\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:18.734696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.734658 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:23:18.741376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:18.741348 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cbda1cc-7dbc678c5f-9mtq7"] Nov 02 15:23:19.242102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.242061 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cda9da2-1b7f-4174-a733-22dcf28b0231" path="/var/lib/kubelet/pods/2cda9da2-1b7f-4174-a733-22dcf28b0231/volumes" Nov 02 15:23:19.314440 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.314408 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" Nov 02 15:23:19.314614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.314441 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn" event={"ID":"d879dd65-0253-490c-a574-c916c01b576e","Type":"ContainerDied","Data":"c0d9b7dafad39853175ef4225b552e8566120b6096e54bf329ed7394ed2638e0"} Nov 02 15:23:19.314614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.314503 4469 scope.go:122] "RemoveContainer" containerID="fb8ddfc31cf3f391022178358fda8af19bff6a2912710430ab8371bbc5060759" Nov 02 15:23:19.362081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.362057 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:19.504869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.503859 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:23:19.597823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:19.597669 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cbda1cc-6cd64c495-85bwn"] Nov 02 15:23:21.242097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:21.242066 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d879dd65-0253-490c-a574-c916c01b576e" path="/var/lib/kubelet/pods/d879dd65-0253-490c-a574-c916c01b576e/volumes" Nov 02 15:23:22.951882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:22.951842 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" podUID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" containerName="payload-processing" containerID="cri-o://f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817" gracePeriod=25 Nov 02 15:23:23.061353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.061314 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" podUID="0d470abe-ca6d-45fe-9384-f232bca0888a" containerName="payload-pre-processing" containerID="cri-o://40fd917433b01c1c34bf2390cce8a0d478220ccfb6c3fe43bc84fb5ca6257e33" gracePeriod=25 Nov 02 15:23:23.256588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.256561 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:23:23.350188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.350157 4469 generic.go:356] "Generic (PLEG): container finished" podID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" containerID="f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817" exitCode=0 Nov 02 15:23:23.350390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.350221 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" Nov 02 15:23:23.350390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.350285 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" event={"ID":"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e","Type":"ContainerDied","Data":"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817"} Nov 02 15:23:23.350390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.350323 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh" event={"ID":"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e","Type":"ContainerDied","Data":"5c6f245941edb88ed5810018ec12049038aa3e68cb066ea897dc0d71b198607b"} Nov 02 15:23:23.350390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.350344 4469 scope.go:122] "RemoveContainer" containerID="f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817" Nov 02 15:23:23.354040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.354001 4469 generic.go:356] "Generic (PLEG): container finished" podID="0d470abe-ca6d-45fe-9384-f232bca0888a" containerID="40fd917433b01c1c34bf2390cce8a0d478220ccfb6c3fe43bc84fb5ca6257e33" exitCode=0 Nov 02 15:23:23.354207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.354188 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" event={"ID":"0d470abe-ca6d-45fe-9384-f232bca0888a","Type":"ContainerDied","Data":"40fd917433b01c1c34bf2390cce8a0d478220ccfb6c3fe43bc84fb5ca6257e33"} Nov 02 15:23:23.380637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.380614 4469 scope.go:122] "RemoveContainer" containerID="f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817" Nov 02 15:23:23.381387 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:23:23.381339 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817\": container with ID starting with f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817 not found: ID does not exist" containerID="f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817" Nov 02 15:23:23.381505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.381380 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817"} err="failed to get container status \"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817\": rpc error: code = NotFound desc = could not find container \"f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817\": container with ID starting with f2360ccbe42f1dcbeb026e84508906d9b74d171e3d35d945d4fffc7d33c25817 not found: ID does not exist" Nov 02 15:23:23.383227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.383203 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:23:23.392426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.392397 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\" (UniqueName: \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\") pod \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " Nov 02 15:23:23.392549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.392484 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\") pod \"0d470abe-ca6d-45fe-9384-f232bca0888a\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " Nov 02 15:23:23.392616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.392563 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\") pod \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\" (UID: \"c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e\") " Nov 02 15:23:23.393214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.392924 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume" pod "0d470abe-ca6d-45fe-9384-f232bca0888a" (UID: "0d470abe-ca6d-45fe-9384-f232bca0888a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:23.393214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.392995 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume" pod "c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" (UID: "c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:23.393467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.393443 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d470abe-ca6d-45fe-9384-f232bca0888a-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:23.393467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.393468 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:23.394760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.394735 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95" pod "c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" (UID: "c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e"). InnerVolumeSpecName "kube-api-access-sxw95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:23.494567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.494501 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\" (UniqueName: \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\") pod \"0d470abe-ca6d-45fe-9384-f232bca0888a\" (UID: \"0d470abe-ca6d-45fe-9384-f232bca0888a\") " Nov 02 15:23:23.495108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.495092 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxw95\" (UniqueName: \"kubernetes.io/projected/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e-kube-api-access-sxw95\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:23.496563 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.496538 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6" pod "0d470abe-ca6d-45fe-9384-f232bca0888a" (UID: "0d470abe-ca6d-45fe-9384-f232bca0888a"). InnerVolumeSpecName "kube-api-access-kmvn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:23.597195 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.597161 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvn6\" (UniqueName: \"kubernetes.io/projected/0d470abe-ca6d-45fe-9384-f232bca0888a-kube-api-access-kmvn6\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:23.773307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.773215 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:23:23.779071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:23.779042 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-748f2e23-7597598dd5-27kfh"] Nov 02 15:23:24.376560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.376527 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" event={"ID":"0d470abe-ca6d-45fe-9384-f232bca0888a","Type":"ContainerDied","Data":"77d045f05426aaa6cafd0c7c5dd2504dcbbd870934ff5e15eaf8db96c6466ed2"} Nov 02 15:23:24.377047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.376578 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t" Nov 02 15:23:24.377047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.376620 4469 scope.go:122] "RemoveContainer" containerID="40fd917433b01c1c34bf2390cce8a0d478220ccfb6c3fe43bc84fb5ca6257e33" Nov 02 15:23:24.462989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.461429 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:24.550639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.550602 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:23:24.599071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:24.599039 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-748f2e23-5dd9db5667-tsk5t"] Nov 02 15:23:25.243439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:25.243406 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d470abe-ca6d-45fe-9384-f232bca0888a" path="/var/lib/kubelet/pods/0d470abe-ca6d-45fe-9384-f232bca0888a/volumes" Nov 02 15:23:25.243951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:25.243929 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" path="/var/lib/kubelet/pods/c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e/volumes" Nov 02 15:23:28.947128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.947094 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:23:28.948038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948016 4469 cpu_manager.go:414] "containerMap: removing container" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948038 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948062 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" containerName="payload-processing" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948070 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" containerName="payload-processing" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948085 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0d470abe-ca6d-45fe-9384-f232bca0888a" containerName="payload-pre-processing" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948093 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d470abe-ca6d-45fe-9384-f232bca0888a" containerName="payload-pre-processing" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948123 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2cda9da2-1b7f-4174-a733-22dcf28b0231" containerName="payload-processing" Nov 02 15:23:28.948155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948130 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cda9da2-1b7f-4174-a733-22dcf28b0231" containerName="payload-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948171 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d879dd65-0253-490c-a574-c916c01b576e" containerName="payload-pre-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948179 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d879dd65-0253-490c-a574-c916c01b576e" containerName="payload-pre-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948353 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="22984f6a-27d0-4558-bbce-8a4ac8f858f2" containerName="istio-proxy" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948369 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6253bf9-f6a3-41ff-a9a1-6ebed1f5d87e" containerName="payload-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948379 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d470abe-ca6d-45fe-9384-f232bca0888a" containerName="payload-pre-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948393 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cda9da2-1b7f-4174-a733-22dcf28b0231" containerName="payload-processing" Nov 02 15:23:28.948490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.948409 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d879dd65-0253-490c-a574-c916c01b576e" containerName="payload-pre-processing" Nov 02 15:23:28.952577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:28.952553 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.007196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.007170 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b87e90-serving-cert\"" Nov 02 15:23:29.060771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.060733 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:23:29.084715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.084682 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.084990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.084970 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.085049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.085008 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx94\" (UniqueName: \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.085049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.085029 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.187111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.187076 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.187316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.187304 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.187384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.187335 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx94\" (UniqueName: \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.187384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.187357 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.187966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.187940 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.189655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.189629 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.189713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.189668 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.209050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.208964 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:29.251044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.251018 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx94\" (UniqueName: \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\") pod \"maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.270364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.270341 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:29.428321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.427729 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:29.652442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:29.652411 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:23:29.653278 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:23:29.653250 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec05cde_9893_43e0_b2c7_1cb4e73849e0.slice/crio-8d2fb15c95b0d6f7fbabd3380512195f25e2011b2f3cd331be7e7cee5d347e54 WatchSource:0}: Error finding container 8d2fb15c95b0d6f7fbabd3380512195f25e2011b2f3cd331be7e7cee5d347e54: Status 404 returned error can't find the container with id 8d2fb15c95b0d6f7fbabd3380512195f25e2011b2f3cd331be7e7cee5d347e54 Nov 02 15:23:30.446585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:30.446542 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" event={"ID":"fec05cde-9893-43e0-b2c7-1cb4e73849e0","Type":"ContainerStarted","Data":"8d2fb15c95b0d6f7fbabd3380512195f25e2011b2f3cd331be7e7cee5d347e54"} Nov 02 15:23:31.461870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:31.461795 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" event={"ID":"fec05cde-9893-43e0-b2c7-1cb4e73849e0","Type":"ContainerStarted","Data":"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5"} Nov 02 15:23:31.462312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:31.461902 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:34.262830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:34.253850 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:36.188946 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:23:36.188898 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:23:36.844085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:36.844031 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" podStartSLOduration=7.990677159 podStartE2EDuration="8.844014335s" totalImagesPullingTime="853.337176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:31.582376965 +0000 UTC m=+1614.465979483" watchObservedRunningTime="2026-11-02 15:23:36.844014335 +0000 UTC m=+1619.727616853" Nov 02 15:23:36.844865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:36.844840 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:23:37.475880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:37.475852 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:23:39.269823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:39.261365 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:39.469262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:39.469227 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:42.256333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:42.256257 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:44.275246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:44.275206 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:45.673277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:45.672927 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:23:45.673277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:45.673014 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:48.695477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.695440 4469 generic.go:356] "Generic (PLEG): container finished" podID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerID="4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b" exitCode=0 Nov 02 15:23:48.696003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.695523 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerDied","Data":"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b"} Nov 02 15:23:48.696003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.695997 4469 scope.go:122] "RemoveContainer" containerID="4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b" Nov 02 15:23:48.697587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.697554 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:48.697706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.697639 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:48.697706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.697686 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:48.698364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.698344 4469 generic.go:356] "Generic (PLEG): container finished" podID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerID="0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc" exitCode=0 Nov 02 15:23:48.698449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.698417 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerDied","Data":"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc"} Nov 02 15:23:48.698869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.698851 4469 scope.go:122] "RemoveContainer" containerID="0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc" Nov 02 15:23:48.700311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.700282 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:48.700382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.700353 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:48.700433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:48.700392 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:23:49.395075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:49.394387 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:49.707897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:49.707794 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerStarted","Data":"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7"} Nov 02 15:23:49.710341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:49.710318 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerStarted","Data":"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd"} Nov 02 15:23:50.093691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.093653 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:23:50.093942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.093875 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:50.093942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.093916 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:50.592883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.592844 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:23:50.593702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.593362 4469 patch_prober.go:29] interesting pod/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:23:50.593702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.593430 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:23:50.684824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.677084 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:50.922195 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:50.922096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:51.093378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:51.093340 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:51.093574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:51.093394 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:51.211465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:51.211421 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:51.593055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:51.592971 4469 patch_prober.go:29] interesting pod/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:23:51.593055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:51.593029 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:23:52.094315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:52.094215 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:52.094315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:52.094292 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:52.216795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:52.216751 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:52.592714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:52.592673 4469 patch_prober.go:29] interesting pod/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:23:52.592925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:52.592734 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:23:53.093266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:53.093227 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:53.093493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:53.093296 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:53.592674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:53.592643 4469 patch_prober.go:29] interesting pod/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:23:53.593164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:53.593138 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:23:54.093563 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.093523 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:54.093749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.093577 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:54.392890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.388100 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:54.577824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.577751 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:23:54.592683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.592645 4469 patch_prober.go:29] interesting pod/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:23:54.593109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:54.592704 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:23:55.093686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:55.093646 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:23:55.093887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:55.093725 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:23:55.596842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:55.596795 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:23:55.598005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:55.597983 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:23:56.098079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:56.098052 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:23:56.098390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:56.098368 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:23:56.099065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:56.099045 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:23:57.760452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:57.760415 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:57.972543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:57.972509 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:58.468513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.468478 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:23:58.468777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.468738 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerName="maas-api" containerID="cri-o://340741cdb90e0c3eafaa7d527245e88948f0752ed21e030b60e64bc593aad3bf" gracePeriod=30 Nov 02 15:23:58.510192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.510128 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.99:8443/health\": dial tcp 10.134.0.99:8443: connect: connection refused" Nov 02 15:23:58.643882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.643844 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:58.829933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.829899 4469 generic.go:356] "Generic (PLEG): container finished" podID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerID="340741cdb90e0c3eafaa7d527245e88948f0752ed21e030b60e64bc593aad3bf" exitCode=0 Nov 02 15:23:58.830365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.829979 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" event={"ID":"6fba9515-49a6-441f-82c1-f6b114b274e9","Type":"ContainerDied","Data":"340741cdb90e0c3eafaa7d527245e88948f0752ed21e030b60e64bc593aad3bf"} Nov 02 15:23:58.830365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.830049 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" event={"ID":"6fba9515-49a6-441f-82c1-f6b114b274e9","Type":"ContainerDied","Data":"bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4"} Nov 02 15:23:58.830365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.830074 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3fbb88869f9713d1377d26f848f545713255af27cc16b0950bcc16447d4ac4" Nov 02 15:23:58.842609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.842580 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:23:58.854819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.854758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:58.929821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.927368 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") pod \"6fba9515-49a6-441f-82c1-f6b114b274e9\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " Nov 02 15:23:58.929821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.927522 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\") pod \"6fba9515-49a6-441f-82c1-f6b114b274e9\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " Nov 02 15:23:58.929821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.927679 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\") pod \"6fba9515-49a6-441f-82c1-f6b114b274e9\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " Nov 02 15:23:58.929821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.927744 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\" (UniqueName: \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\") pod \"6fba9515-49a6-441f-82c1-f6b114b274e9\" (UID: \"6fba9515-49a6-441f-82c1-f6b114b274e9\") " Nov 02 15:23:58.934114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.930227 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca" pod "6fba9515-49a6-441f-82c1-f6b114b274e9" (UID: "6fba9515-49a6-441f-82c1-f6b114b274e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:58.953819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.943988 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls" pod "6fba9515-49a6-441f-82c1-f6b114b274e9" (UID: "6fba9515-49a6-441f-82c1-f6b114b274e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:23:58.953819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.947653 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs" pod "6fba9515-49a6-441f-82c1-f6b114b274e9" (UID: "6fba9515-49a6-441f-82c1-f6b114b274e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:23:58.953819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:58.947795 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f" pod "6fba9515-49a6-441f-82c1-f6b114b274e9" (UID: "6fba9515-49a6-441f-82c1-f6b114b274e9"). InnerVolumeSpecName "kube-api-access-svg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:59.030324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.030298 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:59.030536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.030520 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fba9515-49a6-441f-82c1-f6b114b274e9-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:59.030677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.030665 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fba9515-49a6-441f-82c1-f6b114b274e9-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:59.030770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.030758 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svg8f\" (UniqueName: \"kubernetes.io/projected/6fba9515-49a6-441f-82c1-f6b114b274e9-kube-api-access-svg8f\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:23:59.339119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.339052 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:23:59.340147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.340066 4469 cpu_manager.go:414] "containerMap: removing container" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerName="maas-api" Nov 02 15:23:59.340147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.340089 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerName="maas-api" Nov 02 15:23:59.340421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.340268 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" containerName="maas-api" Nov 02 15:23:59.343157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.343134 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.398831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.395928 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:23:59.439819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.438926 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-de9c4ba4-openshift-default-dockercfg-7b79t\"" Nov 02 15:23:59.460324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460284 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.460482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460343 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.460526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460511 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxkd\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.460635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460615 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.460730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460713 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.460904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.460876 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.461049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.461027 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.461121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.461065 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.461175 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.461154 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.548441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.548386 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:59.563327 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.563287 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.563655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.563637 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.563863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.563843 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.564069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.564055 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.564174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.564160 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.564289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.564274 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.564439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.564423 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.565476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.564658 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.565758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.565709 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxkd\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.565998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.565131 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.566105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.565376 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.567028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.566987 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.568303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.568281 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.568581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.568562 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.572058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.572015 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.606387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.606354 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.627933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.627902 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.628052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.627967 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxkd\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.660036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.660007 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:23:59.808583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.808518 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:23:59.808829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.808775 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" podUID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" containerName="maas-api" containerID="cri-o://e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705" gracePeriod=30 Nov 02 15:23:59.842137 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.842107 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn" Nov 02 15:23:59.974425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:23:59.973617 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:00.084433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.084342 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:24:00.089357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.089321 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5cbda1cc-79b5bcffc9-x4sbn"] Nov 02 15:24:00.121275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.121250 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:24:00.153173 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.153137 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:24:00.160357 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:00.160320 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28912ade_3ae0_4421_b891_da992963b763.slice/crio-30e2cc26d74a23058daf8e9ae3ec4d4d2732953102086cf9d11d612629a70108 WatchSource:0}: Error finding container 30e2cc26d74a23058daf8e9ae3ec4d4d2732953102086cf9d11d612629a70108: Status 404 returned error can't find the container with id 30e2cc26d74a23058daf8e9ae3ec4d4d2732953102086cf9d11d612629a70108 Nov 02 15:24:00.163079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.163040 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:00.163207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.163118 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:00.163207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.163153 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:00.189576 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.189543 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\" (UniqueName: \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\") pod \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " Nov 02 15:24:00.190019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.189995 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\") pod \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " Nov 02 15:24:00.190244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.190211 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") pod \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " Nov 02 15:24:00.190507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.190490 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\") pod \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\" (UID: \"4ac1fc12-6fd3-4657-a91d-7ed11109e952\") " Nov 02 15:24:00.195936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.195853 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7" pod "4ac1fc12-6fd3-4657-a91d-7ed11109e952" (UID: "4ac1fc12-6fd3-4657-a91d-7ed11109e952"). InnerVolumeSpecName "kube-api-access-kmxz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:00.196212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.196188 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca" pod "4ac1fc12-6fd3-4657-a91d-7ed11109e952" (UID: "4ac1fc12-6fd3-4657-a91d-7ed11109e952"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:00.198179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.198150 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls" pod "4ac1fc12-6fd3-4657-a91d-7ed11109e952" (UID: "4ac1fc12-6fd3-4657-a91d-7ed11109e952"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:00.198751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.198725 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs" pod "4ac1fc12-6fd3-4657-a91d-7ed11109e952" (UID: "4ac1fc12-6fd3-4657-a91d-7ed11109e952"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:00.293905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.293579 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:00.293905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.293774 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac1fc12-6fd3-4657-a91d-7ed11109e952-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:00.293905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.293793 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac1fc12-6fd3-4657-a91d-7ed11109e952-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:00.293905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.293872 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmxz7\" (UniqueName: \"kubernetes.io/projected/4ac1fc12-6fd3-4657-a91d-7ed11109e952-kube-api-access-kmxz7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:00.357602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.357440 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:00.637269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.637186 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:00.862133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.862094 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:00.936296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.925177 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerStarted","Data":"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e"} Nov 02 15:24:00.936296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.925240 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerStarted","Data":"30e2cc26d74a23058daf8e9ae3ec4d4d2732953102086cf9d11d612629a70108"} Nov 02 15:24:00.936789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.936724 4469 generic.go:356] "Generic (PLEG): container finished" podID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" containerID="e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705" exitCode=0 Nov 02 15:24:00.936939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.936889 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" event={"ID":"4ac1fc12-6fd3-4657-a91d-7ed11109e952","Type":"ContainerDied","Data":"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705"} Nov 02 15:24:00.937002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.936936 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" event={"ID":"4ac1fc12-6fd3-4657-a91d-7ed11109e952","Type":"ContainerDied","Data":"851ad8446c5cb7ebc9eaa5be914d0a776284bbb884a635d1b587a633bf049a92"} Nov 02 15:24:00.937002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.936968 4469 scope.go:122] "RemoveContainer" containerID="e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705" Nov 02 15:24:00.937184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.937149 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf" Nov 02 15:24:00.993822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.987128 4469 scope.go:122] "RemoveContainer" containerID="e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705" Nov 02 15:24:00.993822 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:00.990269 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705\": container with ID starting with e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705 not found: ID does not exist" containerID="e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705" Nov 02 15:24:00.993822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:00.990313 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705"} err="failed to get container status \"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705\": rpc error: code = NotFound desc = could not find container \"e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705\": container with ID starting with e43714fa70330a127aea7bee1236dca66595bf22458d643514fb9b087685b705 not found: ID does not exist" Nov 02 15:24:01.054637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.054573 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podStartSLOduration=2.05455285 podStartE2EDuration="2.05455285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:01.048565019 +0000 UTC m=+1643.932167540" watchObservedRunningTime="2026-11-02 15:24:01.05455285 +0000 UTC m=+1643.938155370" Nov 02 15:24:01.164423 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.164386 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:24:01.179827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.171312 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-748f2e23-747d54bb78-b8sgf"] Nov 02 15:24:01.252350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.252306 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" path="/var/lib/kubelet/pods/4ac1fc12-6fd3-4657-a91d-7ed11109e952/volumes" Nov 02 15:24:01.253203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.253179 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fba9515-49a6-441f-82c1-f6b114b274e9" path="/var/lib/kubelet/pods/6fba9515-49a6-441f-82c1-f6b114b274e9/volumes" Nov 02 15:24:01.660945 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.660849 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:24:01.666993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.666966 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:24:01.865417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.865377 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:01.946171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.946092 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:24:01.947039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:01.947016 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:24:02.145146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:02.145105 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:02.354314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:02.354280 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:03.037143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:03.037102 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:03.330017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:03.329930 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:24:03.330210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:03.330179 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" containerID="cri-o://12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb" gracePeriod=30 Nov 02 15:24:03.406689 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:03.406638 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:04.530734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:04.530694 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:04.734693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:04.734661 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:05.020546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.020508 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:24:05.021713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.021671 4469 cpu_manager.go:414] "containerMap: removing container" podUID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" containerName="maas-api" Nov 02 15:24:05.021983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.021952 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" containerName="maas-api" Nov 02 15:24:05.022231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.022197 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac1fc12-6fd3-4657-a91d-7ed11109e952" containerName="maas-api" Nov 02 15:24:05.047174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.047139 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.121685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.121642 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:24:05.130790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.130742 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6b2f61-openshift-default-dockercfg-5wgjm\"" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141186 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141350 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141394 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141438 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141478 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141550 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141656 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qck7d\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141741 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.142125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.141849 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.159569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.159533 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:05.258904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.252376 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.258904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.252496 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.260135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.260051 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.271817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.270275 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.271817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.270575 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.271817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.270737 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.271817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.270912 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.271817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.271047 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.272206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.271835 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.274935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.272322 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qck7d\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.274935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.272487 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.274935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.273042 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.274935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.273519 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.274935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.273855 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.280908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.280863 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.292827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.284708 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.335824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.332098 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.335824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.333257 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qck7d\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\") pod \"e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.424530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.424498 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:05.679874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.670938 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:05.728032 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.727995 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:24:05.736665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.736638 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.780909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.780876 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:24:05.791003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.790964 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-559973-openshift-default-dockercfg-zzn5l\"" Nov 02 15:24:05.825833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.825777 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.825866 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826107 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826045 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n296r\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826127 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826223 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826199 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826298 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826418 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826483 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.826562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.826522 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.897178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.897140 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:24:05.900133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.900009 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:05.900268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.900145 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:05.900268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.900231 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:05.900384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.900276 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930245 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930327 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930388 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n296r\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930427 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930473 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930536 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930618 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930651 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.930675 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.931864 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.935919 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.936212 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.936220 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.936470 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.937880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.936698 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.939696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.939669 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:05.994780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:05.994748 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:06.020825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.011322 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerStarted","Data":"5a8e6c4f61b054f382702eaed2a08bc55c5939ace5e4d5b4193fbe555a76aa97"} Nov 02 15:24:06.020825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.014252 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n296r\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\") pod \"e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:06.060473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.060430 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:06.143364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.143320 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:06.249607 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:06.249529 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:24:06.452985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.452951 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:06.563433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.563404 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:24:06.564115 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:06.564088 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11abbca7_0516_43c8_9547_954453117952.slice/crio-50a3c6330f8053d85267869f726d9e1ce504aeb6052fb86d45424f8bb4fcb8c8 WatchSource:0}: Error finding container 50a3c6330f8053d85267869f726d9e1ce504aeb6052fb86d45424f8bb4fcb8c8: Status 404 returned error can't find the container with id 50a3c6330f8053d85267869f726d9e1ce504aeb6052fb86d45424f8bb4fcb8c8 Nov 02 15:24:06.567043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.567008 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:06.567124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.567076 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:06.567124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:06.567112 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:24:07.106467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.104246 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerStarted","Data":"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a"} Nov 02 15:24:07.106467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.104319 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerStarted","Data":"50a3c6330f8053d85267869f726d9e1ce504aeb6052fb86d45424f8bb4fcb8c8"} Nov 02 15:24:07.111071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.110096 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerStarted","Data":"1f8a1060b342e1c335e2602bd1a0b3ce0ed0837ac4dd560430408c2b1156bfe2"} Nov 02 15:24:07.231833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.231137 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podStartSLOduration=2.231119128 podStartE2EDuration="2.231119128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:07.227372924 +0000 UTC m=+1650.110975443" watchObservedRunningTime="2026-11-02 15:24:07.231119128 +0000 UTC m=+1650.114721647" Nov 02 15:24:07.355125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.348824 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podStartSLOduration=3.348779816 podStartE2EDuration="3.348779816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:07.347793218 +0000 UTC m=+1650.231395738" watchObservedRunningTime="2026-11-02 15:24:07.348779816 +0000 UTC m=+1650.232382351" Nov 02 15:24:07.425225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.425136 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:07.432268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.432239 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:08.002207 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:07.999762 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:08.061567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.061530 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:08.067866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.067842 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:08.120524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.120493 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:08.121029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.120683 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:08.121544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.121521 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:24:08.121640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.121587 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:24:08.700720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.700697 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:24:08.806686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806642 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.806926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806709 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.806926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806750 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.806926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806866 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806942 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.806992 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807065 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807073 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:24:08.807332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807118 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807158 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\") pod \"82cdd77b-c55c-4506-b3c4-13365998d0fa\" (UID: \"82cdd77b-c55c-4506-b3c4-13365998d0fa\") " Nov 02 15:24:08.807332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807191 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:24:08.807332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807234 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:24:08.807332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807234 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:08.807582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807389 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:24:08.807986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807964 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.808077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.807989 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.808077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.808003 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.808077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.808027 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.808077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.808040 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cdd77b-c55c-4506-b3c4-13365998d0fa-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.809317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.809291 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:24:08.809449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.809318 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "kube-api-access-vp9ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:08.809497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.809463 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:08.809623 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.809608 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo" pod "82cdd77b-c55c-4506-b3c4-13365998d0fa" (UID: "82cdd77b-c55c-4506-b3c4-13365998d0fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:24:08.909372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.909338 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.909372 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.909369 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.909586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.909383 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp9ns\" (UniqueName: \"kubernetes.io/projected/82cdd77b-c55c-4506-b3c4-13365998d0fa-kube-api-access-vp9ns\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:08.909586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:08.909398 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cdd77b-c55c-4506-b3c4-13365998d0fa-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:09.130205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.130167 4469 generic.go:356] "Generic (PLEG): container finished" podID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerID="12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb" exitCode=0 Nov 02 15:24:09.130664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.130238 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" Nov 02 15:24:09.130664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.130251 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerDied","Data":"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb"} Nov 02 15:24:09.130664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.130356 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg" event={"ID":"82cdd77b-c55c-4506-b3c4-13365998d0fa","Type":"ContainerDied","Data":"d69e719bcc74128c5e67ae320355d491dd357a880cdfc312a6102b905b46e328"} Nov 02 15:24:09.130664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.130392 4469 scope.go:122] "RemoveContainer" containerID="12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb" Nov 02 15:24:09.142932 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.142907 4469 scope.go:122] "RemoveContainer" containerID="635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6" Nov 02 15:24:09.154329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.154305 4469 scope.go:122] "RemoveContainer" containerID="12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb" Nov 02 15:24:09.154590 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:09.154567 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb\": container with ID starting with 12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb not found: ID does not exist" containerID="12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb" Nov 02 15:24:09.154677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.154595 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb"} err="failed to get container status \"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb\": rpc error: code = NotFound desc = could not find container \"12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb\": container with ID starting with 12dd7d4bb082a061aadb94be1ccf9419bcdb47e655092feea53138ca3b1e8aeb not found: ID does not exist" Nov 02 15:24:09.154677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.154613 4469 scope.go:122] "RemoveContainer" containerID="635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6" Nov 02 15:24:09.154874 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:09.154854 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6\": container with ID starting with 635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6 not found: ID does not exist" containerID="635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6" Nov 02 15:24:09.154956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.154879 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6"} err="failed to get container status \"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6\": rpc error: code = NotFound desc = could not find container \"635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6\": container with ID starting with 635172bf7f7ee68e50baf8c98681d62d06566f62d1b5227831cb21ba989522c6 not found: ID does not exist" Nov 02 15:24:09.302833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.302781 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:24:09.306554 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:09.306528 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-748f2e23-openshift-default-588f97c5cf-gqphg"] Nov 02 15:24:10.423074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:10.423043 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:10.660127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:10.660090 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:11.077332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:11.077301 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:11.242375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:11.242344 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" path="/var/lib/kubelet/pods/82cdd77b-c55c-4506-b3c4-13365998d0fa/volumes" Nov 02 15:24:11.284180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:11.284151 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:13.415925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:13.415889 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:13.754757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:13.754725 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:13.968192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:13.968156 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:14.134014 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.133910 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:24:14.135089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135063 4469 cpu_manager.go:414] "containerMap: removing container" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.135089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135089 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.135253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135127 4469 cpu_manager.go:414] "containerMap: removing container" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.135253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135134 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.135329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135300 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.135329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.135323 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cdd77b-c55c-4506-b3c4-13365998d0fa" containerName="istio-proxy" Nov 02 15:24:14.139726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.139697 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.148226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.148191 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.148432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.148410 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmkz\" (UniqueName: \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.193128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.193086 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.225271 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.240022 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-559973\"" Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.240758 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-559973-dockercfg-29cjk\"" Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.250932 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmkz\" (UniqueName: \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.251202 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.259850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.252180 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.412701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.401537 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmkz\" (UniqueName: \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\") pod \"payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.434845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.430932 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:24:14.463894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.452048 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.463894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.458992 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:14.480350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.475260 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:24:14.514734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.514707 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.515106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.515081 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctn7\" (UniqueName: \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.643259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.643220 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctn7\" (UniqueName: \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.644332 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.644299 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.645283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.645252 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.723871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.716752 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctn7\" (UniqueName: \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\") pod \"payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.852902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.852862 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:14.958669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:14.958640 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:24:14.960833 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:14.960535 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5692e221_7431_4fe8_9d4b_9a6afe036821.slice/crio-b6acbece341ec0fd40a5f525e98accc168cc1ef6892ac1b0cfcba6c13a35e5fb WatchSource:0}: Error finding container b6acbece341ec0fd40a5f525e98accc168cc1ef6892ac1b0cfcba6c13a35e5fb: Status 404 returned error can't find the container with id b6acbece341ec0fd40a5f525e98accc168cc1ef6892ac1b0cfcba6c13a35e5fb Nov 02 15:24:15.098205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.098172 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:15.181838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.181777 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" event={"ID":"5692e221-7431-4fe8-9d4b-9a6afe036821","Type":"ContainerStarted","Data":"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4"} Nov 02 15:24:15.182019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.181862 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" event={"ID":"5692e221-7431-4fe8-9d4b-9a6afe036821","Type":"ContainerStarted","Data":"b6acbece341ec0fd40a5f525e98accc168cc1ef6892ac1b0cfcba6c13a35e5fb"} Nov 02 15:24:15.182019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.181998 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:15.262867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.262777 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:24:15.263951 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:15.263925 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb158bdd3_ecc6_460f_96b7_0753f3aa8c1d.slice/crio-08f259c660da097cf2934b150fb56c5ab781245daf3e8743485f96da2c1e464e WatchSource:0}: Error finding container 08f259c660da097cf2934b150fb56c5ab781245daf3e8743485f96da2c1e464e: Status 404 returned error can't find the container with id 08f259c660da097cf2934b150fb56c5ab781245daf3e8743485f96da2c1e464e Nov 02 15:24:15.313777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.313734 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" podStartSLOduration=1.313718425 podStartE2EDuration="1.313718425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:15.308075088 +0000 UTC m=+1658.191677607" watchObservedRunningTime="2026-11-02 15:24:15.313718425 +0000 UTC m=+1658.197320944" Nov 02 15:24:15.325057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.325030 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:15.750984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:15.750948 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:16.191205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.191112 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" event={"ID":"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d","Type":"ContainerStarted","Data":"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d"} Nov 02 15:24:16.191205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.191172 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" event={"ID":"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d","Type":"ContainerStarted","Data":"08f259c660da097cf2934b150fb56c5ab781245daf3e8743485f96da2c1e464e"} Nov 02 15:24:16.191404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.191215 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:16.319549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.319497 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" podStartSLOduration=2.319484199 podStartE2EDuration="2.319484199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:16.315354637 +0000 UTC m=+1659.198957158" watchObservedRunningTime="2026-11-02 15:24:16.319484199 +0000 UTC m=+1659.203086717" Nov 02 15:24:16.513626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.513594 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:16.725997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.725961 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:16.931289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:16.931205 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:19.397002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:19.396541 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:22.381824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:22.381774 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:22.589427 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:22.589393 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:23.634176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:23.634146 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:23.843746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:23.843712 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:24.004239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.004206 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:24:24.008229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.008208 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.061748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.061693 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-6b2f61\"" Nov 02 15:24:24.123595 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.123545 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.123828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.123688 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldr9\" (UniqueName: \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.126210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.126176 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:24:24.165470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.165414 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:24:24.309871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.251289 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.309871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.251435 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldr9\" (UniqueName: \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.309871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.283597 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.320269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.318223 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.370067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.370031 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldr9\" (UniqueName: \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\") pod \"payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.377169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.377136 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:24:24.383309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.383285 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-6b2f61-dockercfg-jbsrl\"" Nov 02 15:24:24.471590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.471539 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.471886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.471611 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8xm\" (UniqueName: \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.574873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.573893 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.574873 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.573964 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8xm\" (UniqueName: \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.578844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.576728 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.625930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.625891 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:24.643229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.641391 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8xm\" (UniqueName: \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\") pod \"payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:24.695361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:24.695330 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:25.000357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.000328 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:25.068609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.067847 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:24:25.073844 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:25.073397 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6397750_8c5c_4ccf_bbf7_e88fbd23eb85.slice/crio-30e21f65852971e2ed2ee8c3523fe94d11c46df24c913e1a173670690f4a6ac7 WatchSource:0}: Error finding container 30e21f65852971e2ed2ee8c3523fe94d11c46df24c913e1a173670690f4a6ac7: Status 404 returned error can't find the container with id 30e21f65852971e2ed2ee8c3523fe94d11c46df24c913e1a173670690f4a6ac7 Nov 02 15:24:25.113484 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:24:25.113366 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e06e67_6bf0_4d82_8fcd_69a8170cf291.slice/crio-2d96403b854245ee5df3e03c538841cc0f8e997d94f41c5b109e2a849183dc18 WatchSource:0}: Error finding container 2d96403b854245ee5df3e03c538841cc0f8e997d94f41c5b109e2a849183dc18: Status 404 returned error can't find the container with id 2d96403b854245ee5df3e03c538841cc0f8e997d94f41c5b109e2a849183dc18 Nov 02 15:24:25.116090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.115426 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:24:25.206056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.206031 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:25.344384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.344306 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" event={"ID":"e9e06e67-6bf0-4d82-8fcd-69a8170cf291","Type":"ContainerStarted","Data":"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f"} Nov 02 15:24:25.344384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.344363 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" event={"ID":"e9e06e67-6bf0-4d82-8fcd-69a8170cf291","Type":"ContainerStarted","Data":"2d96403b854245ee5df3e03c538841cc0f8e997d94f41c5b109e2a849183dc18"} Nov 02 15:24:25.344615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.344415 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:25.346181 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.346149 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" event={"ID":"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85","Type":"ContainerStarted","Data":"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e"} Nov 02 15:24:25.346305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.346208 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" event={"ID":"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85","Type":"ContainerStarted","Data":"30e21f65852971e2ed2ee8c3523fe94d11c46df24c913e1a173670690f4a6ac7"} Nov 02 15:24:25.346305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.346237 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:25.463000 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.462944 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" podStartSLOduration=1.46292782 podStartE2EDuration="1.46292782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:25.460877384 +0000 UTC m=+1668.344479908" watchObservedRunningTime="2026-11-02 15:24:25.46292782 +0000 UTC m=+1668.346530338" Nov 02 15:24:25.477101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:25.477070 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:26.194603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:26.194571 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:24:26.333823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:26.323940 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" podStartSLOduration=3.323917914 podStartE2EDuration="3.323917914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:25.595119194 +0000 UTC m=+1668.478721715" watchObservedRunningTime="2026-11-02 15:24:26.323917914 +0000 UTC m=+1669.207520434" Nov 02 15:24:26.465065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:26.465000 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:26.674388 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:26.674351 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:26.922874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:26.922181 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:27.205398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:27.201614 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:24:30.875164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:30.875126 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:36.006990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:36.006951 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:36.197928 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:36.197891 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:24:36.219697 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:36.219654 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:36.361693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:36.361613 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:24:36.361693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:36.361678 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:24:37.181367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:37.181324 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:37.409128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:37.409092 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:38.801354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:38.801319 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:24:38.917634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:38.917598 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:24:39.247507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:39.247306 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:39.458786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:39.458750 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:39.693543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:39.693434 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:40.101283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:40.101249 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:41.101413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:41.085006 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:41.293707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:41.293671 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:41.803096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:41.803056 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:42.597110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:42.597075 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:42.869067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:42.868981 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:43.802533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:43.802492 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" podUID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" containerName="payload-processing" containerID="cri-o://4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4" gracePeriod=25 Nov 02 15:24:43.918737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:43.918702 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" podUID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" containerName="payload-pre-processing" containerID="cri-o://64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97" gracePeriod=25 Nov 02 15:24:44.088431 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.088403 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:24:44.103106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.103085 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\" (UniqueName: \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\") pod \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " Nov 02 15:24:44.103285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.103272 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\") pod \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\" (UID: \"8080f2a4-75c1-4e88-acd2-c0ba8aa05717\") " Nov 02 15:24:44.104144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.104046 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume" pod "8080f2a4-75c1-4e88-acd2-c0ba8aa05717" (UID: "8080f2a4-75c1-4e88-acd2-c0ba8aa05717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:44.115662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.110728 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt" pod "8080f2a4-75c1-4e88-acd2-c0ba8aa05717" (UID: "8080f2a4-75c1-4e88-acd2-c0ba8aa05717"). InnerVolumeSpecName "kube-api-access-gdlpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:44.190116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.190090 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:24:44.204710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.204686 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\" (UniqueName: \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\") pod \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " Nov 02 15:24:44.204886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.204868 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\") pod \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\" (UID: \"638cbdb8-5061-4918-8324-ad3f8ebfd0dc\") " Nov 02 15:24:44.205269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.205240 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume" pod "638cbdb8-5061-4918-8324-ad3f8ebfd0dc" (UID: "638cbdb8-5061-4918-8324-ad3f8ebfd0dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:44.205728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.205706 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlpt\" (UniqueName: \"kubernetes.io/projected/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-kube-api-access-gdlpt\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:44.205816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.205734 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f2a4-75c1-4e88-acd2-c0ba8aa05717-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:44.205816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.205750 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:44.206777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.206752 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l" pod "638cbdb8-5061-4918-8324-ad3f8ebfd0dc" (UID: "638cbdb8-5061-4918-8324-ad3f8ebfd0dc"). InnerVolumeSpecName "kube-api-access-v4c8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:44.307409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.307379 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4c8l\" (UniqueName: \"kubernetes.io/projected/638cbdb8-5061-4918-8324-ad3f8ebfd0dc-kube-api-access-v4c8l\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:24:44.639675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.639643 4469 generic.go:356] "Generic (PLEG): container finished" podID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" containerID="4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4" exitCode=0 Nov 02 15:24:44.639884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.639684 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" Nov 02 15:24:44.639884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.639770 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" event={"ID":"8080f2a4-75c1-4e88-acd2-c0ba8aa05717","Type":"ContainerDied","Data":"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4"} Nov 02 15:24:44.639884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.639835 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm" event={"ID":"8080f2a4-75c1-4e88-acd2-c0ba8aa05717","Type":"ContainerDied","Data":"9abae77564fdf91d43e24a83f7a39cf55f4fd7e696c6d5874a90de60a5ff8055"} Nov 02 15:24:44.639884 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.639863 4469 scope.go:122] "RemoveContainer" containerID="4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4" Nov 02 15:24:44.643052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.643027 4469 generic.go:356] "Generic (PLEG): container finished" podID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" containerID="64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97" exitCode=0 Nov 02 15:24:44.643164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.643061 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" event={"ID":"638cbdb8-5061-4918-8324-ad3f8ebfd0dc","Type":"ContainerDied","Data":"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97"} Nov 02 15:24:44.643164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.643085 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" Nov 02 15:24:44.643164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.643116 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g" event={"ID":"638cbdb8-5061-4918-8324-ad3f8ebfd0dc","Type":"ContainerDied","Data":"9c13936ff69e21e2ec54359e25f14d9c9eca314f15ab2e6d8bdd018242032d22"} Nov 02 15:24:44.652974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.652956 4469 scope.go:122] "RemoveContainer" containerID="4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4" Nov 02 15:24:44.653326 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:44.653301 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4\": container with ID starting with 4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4 not found: ID does not exist" containerID="4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4" Nov 02 15:24:44.653433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.653333 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4"} err="failed to get container status \"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4\": rpc error: code = NotFound desc = could not find container \"4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4\": container with ID starting with 4e03322024d33c675c728afdcb98f25dc938ca9da5187b2521b68832c5979eb4 not found: ID does not exist" Nov 02 15:24:44.653433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.653354 4469 scope.go:122] "RemoveContainer" containerID="64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97" Nov 02 15:24:44.667269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.667244 4469 scope.go:122] "RemoveContainer" containerID="64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97" Nov 02 15:24:44.667640 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:24:44.667605 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97\": container with ID starting with 64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97 not found: ID does not exist" containerID="64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97" Nov 02 15:24:44.667845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.667784 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97"} err="failed to get container status \"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97\": rpc error: code = NotFound desc = could not find container \"64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97\": container with ID starting with 64f45c4ed97353313f54bc7599612356ea3e8b4ac1774557072c3d75d8a90d97 not found: ID does not exist" Nov 02 15:24:44.806980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.806943 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:24:44.810980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.810951 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be5d503e-7dfdd45c7f-pvx7g"] Nov 02 15:24:44.917059 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.916987 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:24:44.922709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:44.922685 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be5d503e-7cddf876cc-c2skm"] Nov 02 15:24:45.245339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:45.245301 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" path="/var/lib/kubelet/pods/638cbdb8-5061-4918-8324-ad3f8ebfd0dc/volumes" Nov 02 15:24:45.246153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:45.246126 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" path="/var/lib/kubelet/pods/8080f2a4-75c1-4e88-acd2-c0ba8aa05717/volumes" Nov 02 15:24:45.378511 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:45.378467 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:45.690876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:45.625330 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:46.852961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:46.852927 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:47.068773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:47.068727 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:47.452498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:47.452466 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:50.537472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:50.537435 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:50.757414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:50.757282 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:52.571985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:52.571948 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:52.785565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:52.785527 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:57.584606 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:24:57.584530 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:01.953534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:01.953491 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:03.518049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:03.518013 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:03.747055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:03.747022 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:25:03.747255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:03.747096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:03.958993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:03.958886 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:04.413506 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:04.413467 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:06.315403 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:25:06.315365 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:25:06.485417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:06.485381 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:08.859912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:08.859874 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:09.065676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:09.065642 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:09.339121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:09.339072 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:11.420560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:11.420516 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:11.629378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:11.629342 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:11.735192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:11.735139 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:25:11.735488 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:11.735440 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" containerID="cri-o://811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7" gracePeriod=30 Nov 02 15:25:11.852666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:11.852584 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:13.876700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:13.876664 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:14.089525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:14.089491 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:17.137912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.137879 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:25:17.188201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188156 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188237 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188359 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188495 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:17.188577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188555 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188600 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188655 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188737 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188793 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.188930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188868 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\") pod \"2c8c4622-0815-477e-abf4-bfac91d7c620\" (UID: \"2c8c4622-0815-477e-abf4-bfac91d7c620\") " Nov 02 15:25:17.189069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.188998 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:17.189696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.189677 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.189845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.189700 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.190226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.189930 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:17.190769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.190743 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:17.191092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.191054 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:17.192111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.192087 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:17.192219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.192122 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:25:17.192283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.192263 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:17.192344 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.192328 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p" pod "2c8c4622-0815-477e-abf4-bfac91d7c620" (UID: "2c8c4622-0815-477e-abf4-bfac91d7c620"). InnerVolumeSpecName "kube-api-access-9lw7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:17.210105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.210079 4469 generic.go:356] "Generic (PLEG): container finished" podID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerID="811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7" exitCode=0 Nov 02 15:25:17.210209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.210135 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" Nov 02 15:25:17.210209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.210155 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerDied","Data":"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7"} Nov 02 15:25:17.210295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.210242 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s" event={"ID":"2c8c4622-0815-477e-abf4-bfac91d7c620","Type":"ContainerDied","Data":"4420bc449bb99c6ecda19cfe132c80ac8c51a010a911a527a2c47868feda6d80"} Nov 02 15:25:17.210295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.210268 4469 scope.go:122] "RemoveContainer" containerID="811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7" Nov 02 15:25:17.227422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.227401 4469 scope.go:122] "RemoveContainer" containerID="4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b" Nov 02 15:25:17.239268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.239224 4469 scope.go:122] "RemoveContainer" containerID="811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7" Nov 02 15:25:17.239593 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:25:17.239572 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7\": container with ID starting with 811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7 not found: ID does not exist" containerID="811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7" Nov 02 15:25:17.239685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.239602 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7"} err="failed to get container status \"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7\": rpc error: code = NotFound desc = could not find container \"811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7\": container with ID starting with 811c3ef9633598c2b21de25be2e01c36d817ca78ac1c14336284d645918193b7 not found: ID does not exist" Nov 02 15:25:17.239685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.239624 4469 scope.go:122] "RemoveContainer" containerID="4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b" Nov 02 15:25:17.240726 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:25:17.240700 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b\": container with ID starting with 4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b not found: ID does not exist" containerID="4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b" Nov 02 15:25:17.240874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.240735 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b"} err="failed to get container status \"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b\": rpc error: code = NotFound desc = could not find container \"4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b\": container with ID starting with 4cfce5115ae768aff9bcbcf97f611c604d32a3a69fca920b7065682aaea0d83b not found: ID does not exist" Nov 02 15:25:17.291525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291485 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291522 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291540 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291555 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291570 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lw7p\" (UniqueName: \"kubernetes.io/projected/2c8c4622-0815-477e-abf4-bfac91d7c620-kube-api-access-9lw7p\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291586 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8c4622-0815-477e-abf4-bfac91d7c620-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.291810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.291597 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8c4622-0815-477e-abf4-bfac91d7c620-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:25:17.442877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.442710 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:25:17.446986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:17.446952 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be5d503e-openshift-default-f868fdd8f-8ql8s"] Nov 02 15:25:18.820826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:18.817373 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:19.034669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:19.033144 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:19.243296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:19.243267 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" path="/var/lib/kubelet/pods/2c8c4622-0815-477e-abf4-bfac91d7c620/volumes" Nov 02 15:25:23.874874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:23.874841 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:26.692006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:26.691957 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.214703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:28.212086 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.433191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:28.433157 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.988710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:28.988673 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:32.642203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:32.642166 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:32.967166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:32.967129 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:33.177573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:33.177540 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:34.158425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:34.158357 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:35.938760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:35.938721 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:36.378505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:36.378458 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:36.726076 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:25:36.726033 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:25:39.536365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:39.536327 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:42.954131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:42.954087 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:43.523051 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:43.523014 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:44.806822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:44.796725 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:45.044264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:45.044221 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:45.271038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:45.270998 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:45.921900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:45.921861 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:25:50.226945 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:50.220006 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:52.919201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:52.919163 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:54.166304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:54.165919 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:25:54.166304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:54.166010 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:54.168015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:54.167902 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:25:54.168015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:54.167952 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:55.150322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:55.149167 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:58.934560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.934521 4469 generic.go:356] "Generic (PLEG): container finished" podID="28912ade-3ae0-4421-b891-da992963b763" containerID="9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e" exitCode=0 Nov 02 15:25:58.935130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.934594 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerDied","Data":"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e"} Nov 02 15:25:58.935130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.935125 4469 scope.go:122] "RemoveContainer" containerID="9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e" Nov 02 15:25:58.936901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.936870 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:58.937018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.936942 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:58.937018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:58.936979 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.824827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.824711 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:59.974822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.973848 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerStarted","Data":"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97"} Nov 02 15:25:59.977420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.977052 4469 generic.go:356] "Generic (PLEG): container finished" podID="11abbca7-0516-43c8-9547-954453117952" containerID="ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a" exitCode=0 Nov 02 15:25:59.977420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.977123 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerDied","Data":"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a"} Nov 02 15:25:59.977617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.977598 4469 scope.go:122] "RemoveContainer" containerID="ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a" Nov 02 15:25:59.980496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980474 4469 generic.go:356] "Generic (PLEG): container finished" podID="e7defa54-df39-4771-b93b-209602778580" containerID="1f8a1060b342e1c335e2602bd1a0b3ce0ed0837ac4dd560430408c2b1156bfe2" exitCode=0 Nov 02 15:25:59.980593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980520 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerDied","Data":"1f8a1060b342e1c335e2602bd1a0b3ce0ed0837ac4dd560430408c2b1156bfe2"} Nov 02 15:25:59.980719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980685 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.980779 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980764 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.980853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980811 4469 scope.go:122] "RemoveContainer" containerID="1f8a1060b342e1c335e2602bd1a0b3ce0ed0837ac4dd560430408c2b1156bfe2" Nov 02 15:25:59.980853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.980831 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.982237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.982207 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.982308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.982274 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:25:59.982348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:25:59.982313 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:26:00.075949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.066358 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:00.379670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.379583 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:00.665167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.662990 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:26:00.665403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.665372 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:00.665480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.665432 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:00.995346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:00.995302 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerStarted","Data":"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04"} Nov 02 15:26:01.000327 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.000291 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerStarted","Data":"2200a7ff56e036f289c6d8d708399d3eba24fa2fe440dc1bdcef143a90bc1f92"} Nov 02 15:26:01.066825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.064376 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:26:01.066825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.064505 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:01.066825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.064547 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:01.438881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.434021 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:26:01.459824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.441902 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:01.459824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.441965 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:01.663303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.663262 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:01.663581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.663553 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:01.960828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:01.950112 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:26:02.061311 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.061271 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:02.061935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.061901 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:02.427370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.427267 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:02.427752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.427624 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:02.667824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.667185 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:02.667824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:02.667250 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:03.061170 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.061133 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:03.061471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.061439 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:03.434916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.428205 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:03.434916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.428270 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:03.688908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.687927 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:03.688908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:03.687989 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:04.079825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.062181 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:04.079825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.062255 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:04.431879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.428170 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:04.431879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.428247 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:04.661824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.661170 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:04.661824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:04.661234 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:05.064199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.064160 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:05.064491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.064461 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:05.145824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.145367 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:05.428896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.428173 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:05.428896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.428236 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:05.662825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.662248 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:26:05.662825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:05.662316 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:26:06.066826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.064114 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:26:06.066826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.064176 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:26:06.212540 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:26:06.212494 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:26:06.434214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.434115 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:26:06.434214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.434180 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:26:06.677782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.677740 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:26:06.680555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.680527 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:26:06.800734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:06.800701 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:07.026430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.026395 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:07.066655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.066562 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:26:07.066929 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.066908 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:26:07.068407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.068383 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:26:07.431588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.431332 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:26:07.432201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.432137 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:26:07.433628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.433580 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:26:07.856038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:07.855987 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:08.075643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:08.075612 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:09.980055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:09.976581 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:12.032103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:12.032073 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:12.234698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:12.234669 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:15.182440 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:15.182408 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:15.398231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:15.398178 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:15.608633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:15.608584 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:20.596282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:20.596227 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:25.748519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:25.748473 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:31.354730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:31.354646 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:36.215014 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:26:36.214973 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:26:36.890113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:36.890071 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:38.635739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:38.635696 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:38.844787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:38.844733 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:39.027394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:39.027358 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:26:39.027597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:39.027569 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" containerID="cri-o://18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460" gracePeriod=30 Nov 02 15:26:39.091506 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:39.091420 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:39.325277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:39.325190 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:41.388251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:41.388180 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:41.739823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:41.737133 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:42.508687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:42.508619 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:42.717746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:42.717714 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:42.926856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:42.926728 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:43.281189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:43.281161 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:43.949716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:43.949669 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:44.321071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.321045 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:26:44.404823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.404768 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.404987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.404942 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.404987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.404984 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405114 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405161 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405208 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405255 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405313 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405442 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\") pod \"3bb785eb-271a-4042-9225-4417462311a0\" (UID: \"3bb785eb-271a-4042-9225-4417462311a0\") " Nov 02 15:26:44.405524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405474 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:44.405524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405505 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:44.405720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405690 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:44.405720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405705 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:44.405935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.405910 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:44.406331 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.406305 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb785eb-271a-4042-9225-4417462311a0-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.406426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.406336 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.406426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.406351 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.406426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.406365 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.406426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.406380 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.407082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.407058 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:44.407264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.407243 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "kube-api-access-99wgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:44.407386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.407367 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:44.407682 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.407665 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token" pod "3bb785eb-271a-4042-9225-4417462311a0" (UID: "3bb785eb-271a-4042-9225-4417462311a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:44.507996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.507929 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb785eb-271a-4042-9225-4417462311a0-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.507996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.507955 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb785eb-271a-4042-9225-4417462311a0-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.507996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.507965 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wgv\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-kube-api-access-99wgv\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.507996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.507977 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb785eb-271a-4042-9225-4417462311a0-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:26:44.604735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.604703 4469 generic.go:356] "Generic (PLEG): container finished" podID="3bb785eb-271a-4042-9225-4417462311a0" containerID="18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460" exitCode=0 Nov 02 15:26:44.604928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.604767 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" Nov 02 15:26:44.604928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.604794 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" event={"ID":"3bb785eb-271a-4042-9225-4417462311a0","Type":"ContainerDied","Data":"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460"} Nov 02 15:26:44.604928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.604865 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7" event={"ID":"3bb785eb-271a-4042-9225-4417462311a0","Type":"ContainerDied","Data":"205dc6f33f80b8e28834e1a9aea4571ab4183081d5a022d57bab9bf054ad5786"} Nov 02 15:26:44.604928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.604896 4469 scope.go:122] "RemoveContainer" containerID="18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460" Nov 02 15:26:44.628182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.628157 4469 scope.go:122] "RemoveContainer" containerID="18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460" Nov 02 15:26:44.628472 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:26:44.628451 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460\": container with ID starting with 18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460 not found: ID does not exist" containerID="18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460" Nov 02 15:26:44.628548 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.628480 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460"} err="failed to get container status \"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460\": rpc error: code = NotFound desc = could not find container \"18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460\": container with ID starting with 18e78eed2ca1ee79e8b5cacc5ffcf84ab6e5b4dfb3c6121ae094e07bd7e47460 not found: ID does not exist" Nov 02 15:26:44.782148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.782069 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:26:44.789642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:44.789591 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-gn4z7"] Nov 02 15:26:45.245644 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:45.245606 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb785eb-271a-4042-9225-4417462311a0" path="/var/lib/kubelet/pods/3bb785eb-271a-4042-9225-4417462311a0/volumes" Nov 02 15:26:46.609711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:46.609671 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:46.822235 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:46.822197 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:47.591533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:47.591481 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:47.817531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:47.817496 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:48.478635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:48.478577 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:48.766109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:48.766020 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:52.124026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:52.123992 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:52.325615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:52.325566 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:53.308094 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:53.308057 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:53.899155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:53.899118 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:54.105526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:54.105472 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:59.281219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:59.281177 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:59.495727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:59.495694 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:59.703539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:26:59.703424 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.080206 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081363 4469 cpu_manager.go:414] "containerMap: removing container" podUID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" containerName="payload-pre-processing" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081386 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" containerName="payload-pre-processing" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081405 4469 cpu_manager.go:414] "containerMap: removing container" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081414 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081449 4469 cpu_manager.go:414] "containerMap: removing container" podUID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" containerName="payload-processing" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081456 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" containerName="payload-processing" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081505 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081512 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081533 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081540 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081684 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081700 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8c4622-0815-477e-abf4-bfac91d7c620" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081714 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="638cbdb8-5061-4918-8324-ad3f8ebfd0dc" containerName="payload-pre-processing" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081724 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb785eb-271a-4042-9225-4417462311a0" containerName="istio-proxy" Nov 02 15:27:00.088827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.081738 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="8080f2a4-75c1-4e88-acd2-c0ba8aa05717" containerName="payload-processing" Nov 02 15:27:00.185825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.176172 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.185825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.183256 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:27:00.262912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.262120 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.262912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.262526 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqpm\" (UniqueName: \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.278538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.278484 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-885c69-dockercfg-trlsw\"" Nov 02 15:27:00.278706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.278664 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-885c69\"" Nov 02 15:27:00.365186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.365082 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.365669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.365399 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqpm\" (UniqueName: \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.366039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.366012 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.427239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.427207 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqpm\" (UniqueName: \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\") pod \"payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.582678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.582644 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:00.965397 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:27:00.964547 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8dc548_e191_470b_8182_359426652ed6.slice/crio-5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6 WatchSource:0}: Error finding container 5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6: Status 404 returned error can't find the container with id 5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6 Nov 02 15:27:00.965397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:00.965332 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:27:01.088361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:01.088317 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" event={"ID":"8c8dc548-e191-470b-8182-359426652ed6","Type":"ContainerStarted","Data":"95cc8af29a2c16bc5212d63927bd7db62adec3e8aafe2207af6485936fc0869f"} Nov 02 15:27:01.088560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:01.088384 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" event={"ID":"8c8dc548-e191-470b-8182-359426652ed6","Type":"ContainerStarted","Data":"5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6"} Nov 02 15:27:01.088560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:01.088473 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:01.209818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:01.209738 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" podStartSLOduration=2.209713416 podStartE2EDuration="2.209713416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:01.207923079 +0000 UTC m=+1824.091525600" watchObservedRunningTime="2026-11-02 15:27:01.209713416 +0000 UTC m=+1824.093315938" Nov 02 15:27:04.077982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:04.077077 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:06.191433 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:27:06.191392 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:27:06.906602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:06.905886 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:27:07.170709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:07.170610 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:09.195654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:09.195613 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:12.100160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:12.100133 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:27:14.402057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:14.400005 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:19.676925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:19.676844 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:19.884734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:19.884702 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:21.333684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:21.333632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.260115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:22.260070 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.741627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:22.741592 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:25.332301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:25.332270 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:27.037658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:27.037622 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:27.256275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:27.256232 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:30.408400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:30.408351 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:30.636084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:30.636037 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:34.178444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:34.176155 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:35.592626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:35.592593 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:35.875144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:35.875053 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:36.400053 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:36.318458 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:36.497967 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:27:36.497925 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:27:36.906712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:36.906670 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:39.523748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:39.523712 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:27:39.524246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:39.523964 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" containerID="cri-o://653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7" gracePeriod=30 Nov 02 15:27:40.447238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:40.447204 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:40.653696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:40.653663 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:40.790001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:40.789833 4469 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:27:40.790713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:40.789982 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:27:40.952260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:40.951429 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:41.159382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:41.159290 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:43.636266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:43.636228 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:44.268924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:44.268881 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:44.503749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:44.503717 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:45.146519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.146491 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:27:45.195888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.195856 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.195947 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.195991 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196022 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196108 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196134 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:45.196329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196248 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196250 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:45.196329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196263 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:45.196476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196359 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196393 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196433 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:45.196476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196458 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\") pod \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\" (UID: \"754768b0-cfbc-4a2e-b651-e83c61f8edd4\") " Nov 02 15:27:45.196734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.196710 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:45.197273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.197244 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.197406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.197275 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.197406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.197289 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.197406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.197302 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.197406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.197315 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.198489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.198468 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:45.198489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.198468 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "kube-api-access-r2lrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:45.198646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.198490 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:45.198646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.198610 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo" pod "754768b0-cfbc-4a2e-b651-e83c61f8edd4" (UID: "754768b0-cfbc-4a2e-b651-e83c61f8edd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:45.300564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.300482 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.300564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.300510 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.300564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.300533 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lrq\" (UniqueName: \"kubernetes.io/projected/754768b0-cfbc-4a2e-b651-e83c61f8edd4-kube-api-access-r2lrq\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.300564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.300547 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/754768b0-cfbc-4a2e-b651-e83c61f8edd4-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:27:45.633758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.633668 4469 generic.go:356] "Generic (PLEG): container finished" podID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerID="653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7" exitCode=0 Nov 02 15:27:45.633958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.633772 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" Nov 02 15:27:45.633958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.633789 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" event={"ID":"754768b0-cfbc-4a2e-b651-e83c61f8edd4","Type":"ContainerDied","Data":"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7"} Nov 02 15:27:45.633958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.633884 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp" event={"ID":"754768b0-cfbc-4a2e-b651-e83c61f8edd4","Type":"ContainerDied","Data":"c9a73fb76c990b85934923374fba64199b912cb6d827ce731a7edb983f88c82a"} Nov 02 15:27:45.633958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.633916 4469 scope.go:122] "RemoveContainer" containerID="653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7" Nov 02 15:27:45.646967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.646952 4469 scope.go:122] "RemoveContainer" containerID="653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7" Nov 02 15:27:45.647280 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:27:45.647261 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7\": container with ID starting with 653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7 not found: ID does not exist" containerID="653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7" Nov 02 15:27:45.647351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.647290 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7"} err="failed to get container status \"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7\": rpc error: code = NotFound desc = could not find container \"653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7\": container with ID starting with 653061a96f643772c65ed68aa77b7301fad527d97515c8eb96fa2acb38d597d7 not found: ID does not exist" Nov 02 15:27:45.803102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.803072 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:27:45.852600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:45.852563 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-njrmp"] Nov 02 15:27:46.516044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:46.516008 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:47.241049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:47.241012 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" path="/var/lib/kubelet/pods/754768b0-cfbc-4a2e-b651-e83c61f8edd4/volumes" Nov 02 15:27:52.187307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:52.187201 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:53.652638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:53.652603 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:57.517856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:27:57.504837 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:00.819143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:00.819106 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:01.034611 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:01.034559 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:02.596390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:02.596348 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:06.189609 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:28:06.189571 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:28:07.477507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:07.477017 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:07.681677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:07.681647 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:11.585024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:11.584993 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:28:12.388846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:12.382983 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:15.195742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:15.195692 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.329679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:17.329639 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.642820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:17.642709 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.998426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:17.998395 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:22.309238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:22.309185 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:22.527358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:22.527320 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:22.736425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:22.736393 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:27.500436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:27.481935 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:32.412745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:32.412703 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:36.187468 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:28:36.187430 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:28:37.395245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:37.395187 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:38.042567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:38.042530 4469 scope.go:122] "RemoveContainer" containerID="340741cdb90e0c3eafaa7d527245e88948f0752ed21e030b60e64bc593aad3bf" Nov 02 15:28:42.552300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:42.551947 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:44.001835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:44.001778 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:47.497823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:47.493454 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:48.456686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:48.453945 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:48.664914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:48.664872 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:52.479414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:52.479382 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:54.118958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:54.118921 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:57.573817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:28:57.573775 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:02.040281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:02.040226 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:06.188180 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:29:06.188142 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:29:06.729830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:06.728085 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:06.943980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:06.943944 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:07.872711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:07.872673 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:09.660837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:09.650380 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:10.159943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:10.159907 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:10.380497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:10.380459 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:10.862115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:10.862081 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:11.085519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:11.085483 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:11.883393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:11.882587 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:16.815750 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:16.815700 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:22.015582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:22.015435 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:22.238710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:22.238668 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:22.440722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:22.440632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:25.428111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.428075 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:29:25.428636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.428334 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" containerID="cri-o://f847f4a1a048a7a8e959e4ebf0fbede26d92b0e5ba2e1dc3735897fe04d98561" gracePeriod=30 Nov 02 15:29:25.479556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.479523 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:29:25.479821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.479775 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" containerID="cri-o://0b9bbd882e3bed662c0d41eaf86de9a6208a3f6e66f8268e95d923caa76c2d2b" gracePeriod=30 Nov 02 15:29:25.528234 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.528193 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:29:25.529638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:25.529600 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" containerID="cri-o://03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82" gracePeriod=30 Nov 02 15:29:26.430067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:26.430024 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:26.964217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:26.964184 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:27.143002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:27.142960 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:27.143162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:27.143024 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:27.176841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:27.176773 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:28.352204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:28.352103 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:28.362925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:28.362884 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:29:28.363384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:28.363351 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" containerID="cri-o://8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20" gracePeriod=30 Nov 02 15:29:29.057498 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:29.057463 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:29:29.283394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:29.283354 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:29.283551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:29.283420 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:30.065474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.065440 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:30.065886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.065503 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:30.661468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.656680 4469 generic.go:356] "Generic (PLEG): container finished" podID="2312741f-9463-4291-957f-f45ad1ada131" containerID="f847f4a1a048a7a8e959e4ebf0fbede26d92b0e5ba2e1dc3735897fe04d98561" exitCode=0 Nov 02 15:29:30.661468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.656826 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" event={"ID":"2312741f-9463-4291-957f-f45ad1ada131","Type":"ContainerDied","Data":"f847f4a1a048a7a8e959e4ebf0fbede26d92b0e5ba2e1dc3735897fe04d98561"} Nov 02 15:29:30.661468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.659899 4469 generic.go:356] "Generic (PLEG): container finished" podID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerID="0b9bbd882e3bed662c0d41eaf86de9a6208a3f6e66f8268e95d923caa76c2d2b" exitCode=0 Nov 02 15:29:30.661468 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.660015 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" event={"ID":"aa3ba8fa-5235-4866-815b-c1d05be06d90","Type":"ContainerDied","Data":"0b9bbd882e3bed662c0d41eaf86de9a6208a3f6e66f8268e95d923caa76c2d2b"} Nov 02 15:29:30.914271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.914242 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:29:30.928701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.927456 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:29:30.970426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:30.970395 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:29:31.064301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064263 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.064505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064447 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.064505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064484 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.064647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064515 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.064647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064601 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.064647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064624 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.064874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064710 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.064874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064744 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.064874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064795 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064872 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.065060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064927 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.065060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064961 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.064994 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.065060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065023 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065088 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065138 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065190 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065269 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\") pod \"2312741f-9463-4291-957f-f45ad1ada131\" (UID: \"2312741f-9463-4291-957f-f45ad1ada131\") " Nov 02 15:29:31.065326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065322 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\") pod \"aa3ba8fa-5235-4866-815b-c1d05be06d90\" (UID: \"aa3ba8fa-5235-4866-815b-c1d05be06d90\") " Nov 02 15:29:31.065949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.065869 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:31.066809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066110 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.066809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066371 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3ba8fa-5235-4866-815b-c1d05be06d90-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.066809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066396 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.066809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066411 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.066809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066617 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.067115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.066983 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:31.067427 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.067401 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.067633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.067612 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.067846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.067824 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.068025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.068004 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.068204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.068180 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.068651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.068627 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "kube-api-access-tj8ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.069304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.069256 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "kube-api-access-gs8b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.073189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.073145 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:31.073189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.073153 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.073189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.073150 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token" pod "aa3ba8fa-5235-4866-815b-c1d05be06d90" (UID: "aa3ba8fa-5235-4866-815b-c1d05be06d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.073422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.073186 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.073422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.073217 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.074293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.074268 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo" pod "2312741f-9463-4291-957f-f45ad1ada131" (UID: "2312741f-9463-4291-957f-f45ad1ada131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:31.167100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167072 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167111 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167131 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167173 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167244 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167336 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167358 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.167588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167396 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167501 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167588 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167537 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:31.167842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167643 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\") pod \"2fa56acb-43d7-41d3-a4af-88174d4118e4\" (UID: \"2fa56acb-43d7-41d3-a4af-88174d4118e4\") " Nov 02 15:29:31.167842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167666 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.167964 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167857 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.168031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.167965 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.168521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168494 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168525 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168540 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2312741f-9463-4291-957f-f45ad1ada131-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168554 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168567 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8ng\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-kube-api-access-tj8ng\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168581 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168594 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa56acb-43d7-41d3-a4af-88174d4118e4-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168610 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.168625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168623 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168637 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168652 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168665 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2312741f-9463-4291-957f-f45ad1ada131-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168679 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2312741f-9463-4291-957f-f45ad1ada131-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168692 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8b8\" (UniqueName: \"kubernetes.io/projected/2312741f-9463-4291-957f-f45ad1ada131-kube-api-access-gs8b8\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168707 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168720 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168733 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168747 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168760 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.168774 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3ba8fa-5235-4866-815b-c1d05be06d90-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.169763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.169741 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "kube-api-access-6wv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.170000 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.169982 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:31.170129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.170109 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:31.170191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.170130 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token" pod "2fa56acb-43d7-41d3-a4af-88174d4118e4" (UID: "2fa56acb-43d7-41d3-a4af-88174d4118e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:31.276433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.276340 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.276433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.276372 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wv85\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-kube-api-access-6wv85\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.276433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.276390 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.276433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.276405 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa56acb-43d7-41d3-a4af-88174d4118e4-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:31.670697 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.670664 4469 generic.go:356] "Generic (PLEG): container finished" podID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerID="03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82" exitCode=0 Nov 02 15:29:31.670899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.670725 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" Nov 02 15:29:31.670899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.670743 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" event={"ID":"2fa56acb-43d7-41d3-a4af-88174d4118e4","Type":"ContainerDied","Data":"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82"} Nov 02 15:29:31.670899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.670815 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc" event={"ID":"2fa56acb-43d7-41d3-a4af-88174d4118e4","Type":"ContainerDied","Data":"86f9e0b94d74b6658ca229d7c7d37cf79c914e75664a96a868574b7a8095fb65"} Nov 02 15:29:31.670899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.670845 4469 scope.go:122] "RemoveContainer" containerID="03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82" Nov 02 15:29:31.672518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.672499 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" Nov 02 15:29:31.672636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.672535 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252" event={"ID":"2312741f-9463-4291-957f-f45ad1ada131","Type":"ContainerDied","Data":"92181ffdfd1cdba6646d0a995fc9dd8c1cf706cf931afd2dc048d56c66250daa"} Nov 02 15:29:31.674295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.674264 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" event={"ID":"aa3ba8fa-5235-4866-815b-c1d05be06d90","Type":"ContainerDied","Data":"94f4ea748c105d1e19b7a77c0f7d94d10633183d40c61590205483bd0d6a5ff4"} Nov 02 15:29:31.674295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.674287 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z" Nov 02 15:29:31.683217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.683197 4469 scope.go:122] "RemoveContainer" containerID="03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82" Nov 02 15:29:31.683488 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:29:31.683459 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82\": container with ID starting with 03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82 not found: ID does not exist" containerID="03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82" Nov 02 15:29:31.683542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.683488 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82"} err="failed to get container status \"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82\": rpc error: code = NotFound desc = could not find container \"03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82\": container with ID starting with 03d7215435a9072d77457b8d308391ed330eedfdd562a7ff2fde24a891884f82 not found: ID does not exist" Nov 02 15:29:31.683542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.683504 4469 scope.go:122] "RemoveContainer" containerID="f847f4a1a048a7a8e959e4ebf0fbede26d92b0e5ba2e1dc3735897fe04d98561" Nov 02 15:29:31.692890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.692865 4469 scope.go:122] "RemoveContainer" containerID="0b9bbd882e3bed662c0d41eaf86de9a6208a3f6e66f8268e95d923caa76c2d2b" Nov 02 15:29:31.863256 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.855773 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:29:31.904085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:31.904051 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-c6252"] Nov 02 15:29:32.006608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.006575 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:29:32.022435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.022403 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-6dk9z"] Nov 02 15:29:32.135001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.134916 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:29:32.144366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.144341 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-tbbpc"] Nov 02 15:29:32.558153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.558121 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:32.854485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:32.854387 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:33.241100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.241071 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2312741f-9463-4291-957f-f45ad1ada131" path="/var/lib/kubelet/pods/2312741f-9463-4291-957f-f45ad1ada131/volumes" Nov 02 15:29:33.241609 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.241593 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" path="/var/lib/kubelet/pods/2fa56acb-43d7-41d3-a4af-88174d4118e4/volumes" Nov 02 15:29:33.242096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.242080 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" path="/var/lib/kubelet/pods/aa3ba8fa-5235-4866-815b-c1d05be06d90/volumes" Nov 02 15:29:33.268507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.268477 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:33.268677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.268534 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:33.626431 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.626395 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:33.644459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.644434 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:29:33.693052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.693025 4469 generic.go:356] "Generic (PLEG): container finished" podID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerID="8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20" exitCode=0 Nov 02 15:29:33.693194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.693070 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" Nov 02 15:29:33.693194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.693110 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" event={"ID":"aa0894fd-5c6f-43eb-9209-a27cc095fd59","Type":"ContainerDied","Data":"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20"} Nov 02 15:29:33.693194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.693172 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984" event={"ID":"aa0894fd-5c6f-43eb-9209-a27cc095fd59","Type":"ContainerDied","Data":"97d57afaf6b98952b45ff8e038e785ef9d0f3c147685bfefd83d11542caec51f"} Nov 02 15:29:33.693294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.693203 4469 scope.go:122] "RemoveContainer" containerID="8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20" Nov 02 15:29:33.703969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.703949 4469 scope.go:122] "RemoveContainer" containerID="8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20" Nov 02 15:29:33.704277 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:29:33.704252 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20\": container with ID starting with 8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20 not found: ID does not exist" containerID="8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20" Nov 02 15:29:33.704385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.704283 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20"} err="failed to get container status \"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20\": rpc error: code = NotFound desc = could not find container \"8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20\": container with ID starting with 8f16c7a4455c43571319e86eb46aceb4abb129cc46d20fd0382bbf3c24a5ae20 not found: ID does not exist" Nov 02 15:29:33.712746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.712723 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.712874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.712756 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.712943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.712874 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.712943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.712927 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.712999 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:33.713055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713006 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713093 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713114 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:33.713270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713150 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:33.713270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713217 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:33.713270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713221 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713273 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713314 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\") pod \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\" (UID: \"aa0894fd-5c6f-43eb-9209-a27cc095fd59\") " Nov 02 15:29:33.713665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.713642 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:33.714226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.714201 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.714326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.714231 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.714326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.714247 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.714326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.714263 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.714326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.714278 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.715260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.715236 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:33.715339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.715258 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:33.715381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.715331 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:33.715381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.715360 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8" pod "aa0894fd-5c6f-43eb-9209-a27cc095fd59" (UID: "aa0894fd-5c6f-43eb-9209-a27cc095fd59"). InnerVolumeSpecName "kube-api-access-2rbd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:33.819038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.816622 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbd8\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-kube-api-access-2rbd8\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.819038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.816660 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.819038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.816674 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.819038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.816686 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0894fd-5c6f-43eb-9209-a27cc095fd59-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:29:33.838017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:33.837338 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:34.116864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:34.116773 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:29:34.121229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:34.121199 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-cs984"] Nov 02 15:29:34.806610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:34.806576 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:35.014622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:35.014587 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:35.241456 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:35.241423 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" path="/var/lib/kubelet/pods/aa0894fd-5c6f-43eb-9209-a27cc095fd59/volumes" Nov 02 15:29:36.189512 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:29:36.189477 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:29:39.289131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:39.289088 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:40.328734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:40.328696 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:40.568101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:40.568033 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:40.862265 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:40.862223 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:42.481163 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:42.481133 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:44.467485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:44.467443 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:44.711893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:44.711852 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:46.029551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:46.029512 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:47.411662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:47.411619 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:29:47.460876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:47.460842 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:29:47.576734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:47.576695 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:50.083970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:50.083927 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:54.344646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:54.344610 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:54.460714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:54.460678 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:29:54.576508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:54.576468 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:29:54.876190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:54.874842 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.097006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:55.096966 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.305270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:55.305234 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.667015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:55.666927 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:29:55.667488 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:55.667201 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" containerID="cri-o://892d5dc3dd50f01cc1785b5210fd420fafc0bb9a71a3f7d18d8ae8b3f5c91bf2" gracePeriod=30 Nov 02 15:29:57.068938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:57.068903 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:29:57.179581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:57.179545 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:29:57.528821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:57.523201 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:57.743014 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:57.742982 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:58.742820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:58.742711 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:58.742820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:58.742790 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:59.461596 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:59.461558 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" containerID="cri-o://c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b" gracePeriod=25 Nov 02 15:29:59.577033 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:59.576987 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" containerID="cri-o://cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4" gracePeriod=25 Nov 02 15:29:59.580038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:59.580009 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:29:59.688595 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:29:59.688558 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:30:00.011051 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.011010 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:00.223151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.223122 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:00.224142 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224107 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" Nov 02 15:30:00.224142 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224139 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224155 4469 cpu_manager.go:414] "containerMap: removing container" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224160 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224168 4469 cpu_manager.go:414] "containerMap: removing container" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224173 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224199 4469 cpu_manager.go:414] "containerMap: removing container" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224204 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224275 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" Nov 02 15:30:00.224342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224282 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" Nov 02 15:30:00.224580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224397 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2312741f-9463-4291-957f-f45ad1ada131" containerName="istio-proxy" Nov 02 15:30:00.224580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224409 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa56acb-43d7-41d3-a4af-88174d4118e4" containerName="istio-proxy" Nov 02 15:30:00.224580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224416 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3ba8fa-5235-4866-815b-c1d05be06d90" containerName="istio-proxy" Nov 02 15:30:00.224580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224422 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0894fd-5c6f-43eb-9209-a27cc095fd59" containerName="istio-proxy" Nov 02 15:30:00.224580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.224431 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="754768b0-cfbc-4a2e-b651-e83c61f8edd4" containerName="istio-proxy" Nov 02 15:30:00.232326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.232302 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:00.232486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.232374 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.238066 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.238044 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.292680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.292570 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:00.301055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.301026 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:00.301692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.301613 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.313694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.313537 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:00.314157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.314138 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.322617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.322033 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:00.322617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.322079 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:00.322617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.322180 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.328950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.328931 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:00.342210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.342188 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:00.342997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.342972 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.355715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.355688 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.359006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.357995 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kbn\" (UniqueName: \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.359006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.358157 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.359006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.358692 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.359006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.358939 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc7f\" (UniqueName: \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.363602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.361552 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:00.368244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.365744 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:30:00.373495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.372890 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:00.373495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.372999 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.377702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.377671 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:00.426146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.426120 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:00.436862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.436791 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:00.442180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.441717 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:00.447569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.447536 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:00.460585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460457 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.460585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460529 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.460755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460618 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.460755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460656 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc7f\" (UniqueName: \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.460755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460695 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxlm\" (UniqueName: \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.460755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460711 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlct\" (UniqueName: \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.460755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460737 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khh2f\" (UniqueName: \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.460944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460857 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzhb\" (UniqueName: \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.460944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460895 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqz92\" (UniqueName: \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.461008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460946 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kbn\" (UniqueName: \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.461008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.460987 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.461078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461057 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.461222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461203 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.461483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461448 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6hc8\" (UniqueName: \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.461586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461503 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.461655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461599 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.461877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.461849 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.462181 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.462161 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.534443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.534400 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc7f\" (UniqueName: \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.540627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.540572 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kbn\" (UniqueName: \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\") pod \"maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.549790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.549213 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:00.555592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.555567 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:00.563312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563285 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6hc8\" (UniqueName: \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.563422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563321 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.563619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563593 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.563708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563669 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.563765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563733 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.563829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563791 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxlm\" (UniqueName: \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.563893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563832 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlct\" (UniqueName: \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.563893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563874 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khh2f\" (UniqueName: \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.564009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.563975 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzhb\" (UniqueName: \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.564071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.564012 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqz92\" (UniqueName: \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.564128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.564091 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.564381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.564184 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.564381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.564200 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.564552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.564473 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.565103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.565083 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.565251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.565188 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.565398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.565379 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.565998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.565971 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.626142 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.626111 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxlm\" (UniqueName: \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\") pod \"maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.633687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.633244 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqz92\" (UniqueName: \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.634011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.633910 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlct\" (UniqueName: \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\") pod \"maas-api-key-cleanup-29893890-6g76t\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.634638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.634614 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6hc8\" (UniqueName: \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.637387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.637355 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzhb\" (UniqueName: \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.637780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.637755 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khh2f\" (UniqueName: \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.658145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.650556 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:00.661499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.661392 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:00.696862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.696833 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:00.735846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.733488 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:00.752540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.752509 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:00.929117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.928853 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:00.960122 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:30:00.960092 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ba386c_d24f_480d_ad3b_5f66d742c4c4.slice/crio-6bace28caa424c12e71f3200873952f16394c2353de1e9b4f4e029f9726d4c92 WatchSource:0}: Error finding container 6bace28caa424c12e71f3200873952f16394c2353de1e9b4f4e029f9726d4c92: Status 404 returned error can't find the container with id 6bace28caa424c12e71f3200873952f16394c2353de1e9b4f4e029f9726d4c92 Nov 02 15:30:00.960861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.960738 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:00.999914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:00.999886 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:01.016129 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:30:01.016100 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda298d4d4_4178_4ee1_995b_707345440cf4.slice/crio-ea7218a177099c218ab42d843839a57ffba339c7c193c51fb4a8a7343edd2334 WatchSource:0}: Error finding container ea7218a177099c218ab42d843839a57ffba339c7c193c51fb4a8a7343edd2334: Status 404 returned error can't find the container with id ea7218a177099c218ab42d843839a57ffba339c7c193c51fb4a8a7343edd2334 Nov 02 15:30:01.075902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.075858 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerStarted","Data":"6bace28caa424c12e71f3200873952f16394c2353de1e9b4f4e029f9726d4c92"} Nov 02 15:30:01.078279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.078249 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerStarted","Data":"ea7218a177099c218ab42d843839a57ffba339c7c193c51fb4a8a7343edd2334"} Nov 02 15:30:01.080369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.080344 4469 generic.go:356] "Generic (PLEG): container finished" podID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerID="892d5dc3dd50f01cc1785b5210fd420fafc0bb9a71a3f7d18d8ae8b3f5c91bf2" exitCode=0 Nov 02 15:30:01.080650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.080512 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" event={"ID":"11c1ebc6-79ba-4e00-886c-e044eec5ecb4","Type":"ContainerDied","Data":"892d5dc3dd50f01cc1785b5210fd420fafc0bb9a71a3f7d18d8ae8b3f5c91bf2"} Nov 02 15:30:01.140535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.140509 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508338 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508382 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508419 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508465 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508515 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508665 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.510179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.508786 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.555906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.552502 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:01.555906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.552794 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:01.575088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.574394 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:01.576570 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.575839 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:01.578216 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.577589 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.578216 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.577662 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\") pod \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\" (UID: \"11c1ebc6-79ba-4e00-886c-e044eec5ecb4\") " Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.578407 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.578433 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.578443 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.584461 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.584631 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:01.604194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.585057 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:01.632077 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:30:01.632048 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6b9172_a97d_49a0_a6ff_f4d126ad54d3.slice/crio-754e9630d44fbe3b298b863b605e3d576359ffc326e265724b0b06e464f03c5f WatchSource:0}: Error finding container 754e9630d44fbe3b298b863b605e3d576359ffc326e265724b0b06e464f03c5f: Status 404 returned error can't find the container with id 754e9630d44fbe3b298b863b605e3d576359ffc326e265724b0b06e464f03c5f Nov 02 15:30:01.638246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.637218 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:01.638246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.637263 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:01.638246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.637290 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:01.640589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.640570 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:01.645484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.645459 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:01.650316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.649854 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:01.699850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.699726 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.701171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.701130 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.701334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.701317 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.755938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.755385 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "kube-api-access-kwzpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:01.755938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.755475 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:30:01.755938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.755528 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy" pod "11c1ebc6-79ba-4e00-886c-e044eec5ecb4" (UID: "11c1ebc6-79ba-4e00-886c-e044eec5ecb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:01.758617 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:30:01.758592 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847b00d3_3f96_4521_936c_e47dde2fe6e8.slice/crio-611aa453c134033ccbf5a1d97fd0d99fdc8dae1a1d6a01ee8fb50e40aa58b07c WatchSource:0}: Error finding container 611aa453c134033ccbf5a1d97fd0d99fdc8dae1a1d6a01ee8fb50e40aa58b07c: Status 404 returned error can't find the container with id 611aa453c134033ccbf5a1d97fd0d99fdc8dae1a1d6a01ee8fb50e40aa58b07c Nov 02 15:30:01.790850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.790775 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:01.802776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.802745 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.802776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.802777 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzpw\" (UniqueName: \"kubernetes.io/projected/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-kube-api-access-kwzpw\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:01.802975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:01.802787 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c1ebc6-79ba-4e00-886c-e044eec5ecb4-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:02.069876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.069792 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" podUID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" containerName="payload-processing" containerID="cri-o://719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d" gracePeriod=25 Nov 02 15:30:02.089616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.089582 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerStarted","Data":"15e9af30c2bdc834fbfbcff19234922928a4e2d9f32a9627b820ed841fca0b47"} Nov 02 15:30:02.091038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.091014 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" Nov 02 15:30:02.091038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.091026 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr" event={"ID":"11c1ebc6-79ba-4e00-886c-e044eec5ecb4","Type":"ContainerDied","Data":"88e466a4e16378ab0862fe9270869f2a0caa284f1713267843c67b223de18028"} Nov 02 15:30:02.091237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.091087 4469 scope.go:122] "RemoveContainer" containerID="892d5dc3dd50f01cc1785b5210fd420fafc0bb9a71a3f7d18d8ae8b3f5c91bf2" Nov 02 15:30:02.092904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.092870 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerStarted","Data":"20888799b22ef66b2ed37932174797b350370d8eb7ba42e4d7f1a2056460f978"} Nov 02 15:30:02.096153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.096127 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" event={"ID":"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3","Type":"ContainerStarted","Data":"754e9630d44fbe3b298b863b605e3d576359ffc326e265724b0b06e464f03c5f"} Nov 02 15:30:02.097678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.097658 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerStarted","Data":"4ea9dd0e160b0bd5239b42158e985c28dd052efd3fb65e1b76c7a48cd7b57d99"} Nov 02 15:30:02.105430 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.105402 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerStarted","Data":"58392d213bafe271e6c2446faf1d082eb873fe16ede7954a14bb876941246a8c"} Nov 02 15:30:02.106994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.106971 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerStarted","Data":"611aa453c134033ccbf5a1d97fd0d99fdc8dae1a1d6a01ee8fb50e40aa58b07c"} Nov 02 15:30:02.108651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.108627 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerStarted","Data":"fd4ab9297cbde264cbe1176eed280afc06189c907e5cb111742a457198189c76"} Nov 02 15:30:02.109841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.109818 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerStarted","Data":"5696dd9fc504c305045be23f095776ab9bec6ac73a582ed2a048d4852571c0ee"} Nov 02 15:30:02.180202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.180144 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" podUID="5692e221-7431-4fe8-9d4b-9a6afe036821" containerName="payload-pre-processing" containerID="cri-o://a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4" gracePeriod=25 Nov 02 15:30:02.265702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.265637 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" podStartSLOduration=2.265616761 podStartE2EDuration="2.265616761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.263673567 +0000 UTC m=+2005.147276111" watchObservedRunningTime="2026-11-02 15:30:02.265616761 +0000 UTC m=+2005.149219280" Nov 02 15:30:02.394065 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.394007 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:30:02.412504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.412466 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-5sfdr"] Nov 02 15:30:02.439930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.439192 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" containerID="cri-o://cd28e19ba6ee284563ae16e43c18e2652b9637e276f091d22afb07bc6b1e8ad8" gracePeriod=45 Nov 02 15:30:02.499264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.499204 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" podStartSLOduration=2.499184763 podStartE2EDuration="2.499184763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.498198366 +0000 UTC m=+2005.381800886" watchObservedRunningTime="2026-11-02 15:30:02.499184763 +0000 UTC m=+2005.382787283" Nov 02 15:30:02.523458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.523429 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:30:02.618562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.618530 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\" (UniqueName: \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\") pod \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " Nov 02 15:30:02.618721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.618663 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\") pod \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\" (UID: \"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d\") " Nov 02 15:30:02.619433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.619131 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume" pod "b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" (UID: "b158bdd3-ecc6-460f-96b7-0753f3aa8c1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:02.619785 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.619759 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:02.621245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.621219 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7" pod "b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" (UID: "b158bdd3-ecc6-460f-96b7-0753f3aa8c1d"). InnerVolumeSpecName "kube-api-access-gctn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:02.627736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.627656 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:30:02.751948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.751854 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\") pod \"5692e221-7431-4fe8-9d4b-9a6afe036821\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " Nov 02 15:30:02.752480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.752096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:02.752685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.752659 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume" pod "5692e221-7431-4fe8-9d4b-9a6afe036821" (UID: "5692e221-7431-4fe8-9d4b-9a6afe036821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:02.752842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.752822 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\" (UniqueName: \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\") pod \"5692e221-7431-4fe8-9d4b-9a6afe036821\" (UID: \"5692e221-7431-4fe8-9d4b-9a6afe036821\") " Nov 02 15:30:02.753625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.753605 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctn7\" (UniqueName: \"kubernetes.io/projected/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d-kube-api-access-gctn7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:02.754008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.753992 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5692e221-7431-4fe8-9d4b-9a6afe036821-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:02.850462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.850396 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz" pod "5692e221-7431-4fe8-9d4b-9a6afe036821" (UID: "5692e221-7431-4fe8-9d4b-9a6afe036821"). InnerVolumeSpecName "kube-api-access-8xmkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:02.857719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.857535 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" containerID="cri-o://8b1989bca5cda958f4ba084deb2e140f6d49ae8936766f860f361fe2645b7180" gracePeriod=45 Nov 02 15:30:02.858628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.858590 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmkz\" (UniqueName: \"kubernetes.io/projected/5692e221-7431-4fe8-9d4b-9a6afe036821-kube-api-access-8xmkz\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:02.961816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:02.961769 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:03.128918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.128763 4469 generic.go:356] "Generic (PLEG): container finished" podID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerID="33d1c32e08c0fb684961a6feeef2a44e7d05408786ca287d52bc47cdb4b3986e" exitCode=60 Nov 02 15:30:03.128918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.128836 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerDied","Data":"33d1c32e08c0fb684961a6feeef2a44e7d05408786ca287d52bc47cdb4b3986e"} Nov 02 15:30:03.129382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.129303 4469 scope.go:122] "RemoveContainer" containerID="33d1c32e08c0fb684961a6feeef2a44e7d05408786ca287d52bc47cdb4b3986e" Nov 02 15:30:03.131098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.131073 4469 generic.go:356] "Generic (PLEG): container finished" podID="5692e221-7431-4fe8-9d4b-9a6afe036821" containerID="a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4" exitCode=0 Nov 02 15:30:03.131209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.131118 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" Nov 02 15:30:03.131209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.131155 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" event={"ID":"5692e221-7431-4fe8-9d4b-9a6afe036821","Type":"ContainerDied","Data":"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4"} Nov 02 15:30:03.131387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.131220 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw" event={"ID":"5692e221-7431-4fe8-9d4b-9a6afe036821","Type":"ContainerDied","Data":"b6acbece341ec0fd40a5f525e98accc168cc1ef6892ac1b0cfcba6c13a35e5fb"} Nov 02 15:30:03.131387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.131251 4469 scope.go:122] "RemoveContainer" containerID="a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4" Nov 02 15:30:03.133303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.133279 4469 generic.go:356] "Generic (PLEG): container finished" podID="a298d4d4-4178-4ee1-995b-707345440cf4" containerID="fd4ab9297cbde264cbe1176eed280afc06189c907e5cb111742a457198189c76" exitCode=60 Nov 02 15:30:03.133416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.133351 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerDied","Data":"fd4ab9297cbde264cbe1176eed280afc06189c907e5cb111742a457198189c76"} Nov 02 15:30:03.133726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.133703 4469 scope.go:122] "RemoveContainer" containerID="fd4ab9297cbde264cbe1176eed280afc06189c907e5cb111742a457198189c76" Nov 02 15:30:03.138736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.138709 4469 generic.go:356] "Generic (PLEG): container finished" podID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerID="16326ba086861a6149f17b3d7dfef2e1b80d7c4e91f9ece4a028a4a673a3e131" exitCode=60 Nov 02 15:30:03.138879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.138792 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerDied","Data":"16326ba086861a6149f17b3d7dfef2e1b80d7c4e91f9ece4a028a4a673a3e131"} Nov 02 15:30:03.139042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.139026 4469 scope.go:122] "RemoveContainer" containerID="16326ba086861a6149f17b3d7dfef2e1b80d7c4e91f9ece4a028a4a673a3e131" Nov 02 15:30:03.141296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.141273 4469 generic.go:356] "Generic (PLEG): container finished" podID="d759e268-21dd-4530-9a59-780148841f93" containerID="abcf8e81d0d7c0b8f236fcea438c2916fc75a097e6313cfd2341f48e857eb5e4" exitCode=60 Nov 02 15:30:03.141373 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.141357 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerDied","Data":"abcf8e81d0d7c0b8f236fcea438c2916fc75a097e6313cfd2341f48e857eb5e4"} Nov 02 15:30:03.142346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.142324 4469 scope.go:122] "RemoveContainer" containerID="abcf8e81d0d7c0b8f236fcea438c2916fc75a097e6313cfd2341f48e857eb5e4" Nov 02 15:30:03.144589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.144479 4469 generic.go:356] "Generic (PLEG): container finished" podID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerID="cd28e19ba6ee284563ae16e43c18e2652b9637e276f091d22afb07bc6b1e8ad8" exitCode=0 Nov 02 15:30:03.144589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.144543 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" event={"ID":"1fa67505-1d76-438f-b61d-81b0bd53f704","Type":"ContainerDied","Data":"cd28e19ba6ee284563ae16e43c18e2652b9637e276f091d22afb07bc6b1e8ad8"} Nov 02 15:30:03.144589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.144576 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" event={"ID":"1fa67505-1d76-438f-b61d-81b0bd53f704","Type":"ContainerDied","Data":"4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1"} Nov 02 15:30:03.144770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.144592 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e96210b42a5c05e4f75a43c1a49e059c8e597fedc47f720b24c1b075faaffb1" Nov 02 15:30:03.146341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.146327 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" Nov 02 15:30:03.146414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.146290 4469 generic.go:356] "Generic (PLEG): container finished" podID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" containerID="719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d" exitCode=0 Nov 02 15:30:03.146525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.146507 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" event={"ID":"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d","Type":"ContainerDied","Data":"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d"} Nov 02 15:30:03.146578 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.146550 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc" event={"ID":"b158bdd3-ecc6-460f-96b7-0753f3aa8c1d","Type":"ContainerDied","Data":"08f259c660da097cf2934b150fb56c5ab781245daf3e8743485f96da2c1e464e"} Nov 02 15:30:03.148083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.148063 4469 generic.go:356] "Generic (PLEG): container finished" podID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerID="20888799b22ef66b2ed37932174797b350370d8eb7ba42e4d7f1a2056460f978" exitCode=60 Nov 02 15:30:03.148151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.148136 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerDied","Data":"20888799b22ef66b2ed37932174797b350370d8eb7ba42e4d7f1a2056460f978"} Nov 02 15:30:03.148448 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.148431 4469 scope.go:122] "RemoveContainer" containerID="20888799b22ef66b2ed37932174797b350370d8eb7ba42e4d7f1a2056460f978" Nov 02 15:30:03.150299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.150278 4469 generic.go:356] "Generic (PLEG): container finished" podID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerID="facb5e751ad95b23e7e9b97593570a926d017b4b3aeb3f4f66ec64a018998d82" exitCode=60 Nov 02 15:30:03.150412 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.150346 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" event={"ID":"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3","Type":"ContainerDied","Data":"facb5e751ad95b23e7e9b97593570a926d017b4b3aeb3f4f66ec64a018998d82"} Nov 02 15:30:03.150580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.150564 4469 scope.go:122] "RemoveContainer" containerID="facb5e751ad95b23e7e9b97593570a926d017b4b3aeb3f4f66ec64a018998d82" Nov 02 15:30:03.152411 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.152393 4469 generic.go:356] "Generic (PLEG): container finished" podID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerID="8b1989bca5cda958f4ba084deb2e140f6d49ae8936766f860f361fe2645b7180" exitCode=0 Nov 02 15:30:03.152527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.152459 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" event={"ID":"a2dbd84f-4471-454d-89d3-48f756e99e51","Type":"ContainerDied","Data":"8b1989bca5cda958f4ba084deb2e140f6d49ae8936766f860f361fe2645b7180"} Nov 02 15:30:03.154953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.154587 4469 generic.go:356] "Generic (PLEG): container finished" podID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerID="a7e9fd1dd2868324bacf5e51bfa4f24acdd9bf535759d03319c976435d378194" exitCode=60 Nov 02 15:30:03.154953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.154854 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerDied","Data":"a7e9fd1dd2868324bacf5e51bfa4f24acdd9bf535759d03319c976435d378194"} Nov 02 15:30:03.155436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.155412 4469 scope.go:122] "RemoveContainer" containerID="a7e9fd1dd2868324bacf5e51bfa4f24acdd9bf535759d03319c976435d378194" Nov 02 15:30:03.159417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.159058 4469 generic.go:356] "Generic (PLEG): container finished" podID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerID="359807ce95371a867be1a6ced505c0b3eba3f210d87bfcec30535888c40ef377" exitCode=60 Nov 02 15:30:03.159417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.159102 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerDied","Data":"359807ce95371a867be1a6ced505c0b3eba3f210d87bfcec30535888c40ef377"} Nov 02 15:30:03.159417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.159401 4469 scope.go:122] "RemoveContainer" containerID="359807ce95371a867be1a6ced505c0b3eba3f210d87bfcec30535888c40ef377" Nov 02 15:30:03.197975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.197956 4469 scope.go:122] "RemoveContainer" containerID="a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4" Nov 02 15:30:03.198266 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:03.198239 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4\": container with ID starting with a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4 not found: ID does not exist" containerID="a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4" Nov 02 15:30:03.198360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.198278 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4"} err="failed to get container status \"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4\": rpc error: code = NotFound desc = could not find container \"a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4\": container with ID starting with a17bf41455df4144b8f692e40f36de56fca4c5c0e18f174340fdf99edb1304a4 not found: ID does not exist" Nov 02 15:30:03.198360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.198301 4469 scope.go:122] "RemoveContainer" containerID="719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d" Nov 02 15:30:03.198479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.198400 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:30:03.227825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.225976 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:30:03.232121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.231538 4469 scope.go:122] "RemoveContainer" containerID="719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d" Nov 02 15:30:03.233773 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:03.233750 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d\": container with ID starting with 719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d not found: ID does not exist" containerID="719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d" Nov 02 15:30:03.233915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.233891 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d"} err="failed to get container status \"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d\": rpc error: code = NotFound desc = could not find container \"719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d\": container with ID starting with 719042c06c9c786687a61c29e37bdb941f9161fcad44239babab9981a4c2700d not found: ID does not exist" Nov 02 15:30:03.248123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.248079 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" path="/var/lib/kubelet/pods/11c1ebc6-79ba-4e00-886c-e044eec5ecb4/volumes" Nov 02 15:30:03.264796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.264773 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.264940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.264926 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.265004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.264966 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.265004 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.264996 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.265135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265022 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265042 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\" (UniqueName: \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265078 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265123 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.265346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265199 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\" (UniqueName: \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\") pod \"a2dbd84f-4471-454d-89d3-48f756e99e51\" (UID: \"a2dbd84f-4471-454d-89d3-48f756e99e51\") " Nov 02 15:30:03.265346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265203 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.265346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265281 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265291 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.265557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265377 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265422 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\") pod \"1fa67505-1d76-438f-b61d-81b0bd53f704\" (UID: \"1fa67505-1d76-438f-b61d-81b0bd53f704\") " Nov 02 15:30:03.265666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265371 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.265666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265393 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.265666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265575 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.266155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.265886 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266392 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266538 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266561 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266576 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266598 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.266730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.266610 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.327876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.325213 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.327876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.325882 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "kube-api-access-s67cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:03.327876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.325957 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:03.327876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.326046 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "kube-api-access-zg6bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:03.328191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.328172 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs" pod "a2dbd84f-4471-454d-89d3-48f756e99e51" (UID: "a2dbd84f-4471-454d-89d3-48f756e99e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:30:03.328248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.328175 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs" pod "1fa67505-1d76-438f-b61d-81b0bd53f704" (UID: "1fa67505-1d76-438f-b61d-81b0bd53f704"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:30:03.367723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367691 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa67505-1d76-438f-b61d-81b0bd53f704-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.367723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367721 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2dbd84f-4471-454d-89d3-48f756e99e51-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.367925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367731 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2dbd84f-4471-454d-89d3-48f756e99e51-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.367925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367739 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa67505-1d76-438f-b61d-81b0bd53f704-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.367925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367748 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6bz\" (UniqueName: \"kubernetes.io/projected/1fa67505-1d76-438f-b61d-81b0bd53f704-kube-api-access-zg6bz\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.367925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.367758 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67cz\" (UniqueName: \"kubernetes.io/projected/a2dbd84f-4471-454d-89d3-48f756e99e51-kube-api-access-s67cz\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:03.484141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.484113 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:30:03.487740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.487710 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-559973-7b9b654d6f-pptpc"] Nov 02 15:30:03.593105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.593037 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:30:03.596596 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.596572 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-559973-c967685f-ls5fw"] Nov 02 15:30:03.957457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:03.957361 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:30:04.162986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.162895 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:04.191830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.185848 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:04.198864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.198715 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:30:04.236083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.235449 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerStarted","Data":"76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3"} Nov 02 15:30:04.240416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.240383 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerStarted","Data":"f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc"} Nov 02 15:30:04.244098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.244071 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" event={"ID":"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3","Type":"ContainerStarted","Data":"0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81"} Nov 02 15:30:04.251998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.248399 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" event={"ID":"a2dbd84f-4471-454d-89d3-48f756e99e51","Type":"ContainerDied","Data":"11297ad057b0482eea2101b04ca98b48ae21297a43f19d12cc240cccf927c2e1"} Nov 02 15:30:04.251998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.248458 4469 scope.go:122] "RemoveContainer" containerID="8b1989bca5cda958f4ba084deb2e140f6d49ae8936766f860f361fe2645b7180" Nov 02 15:30:04.251998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.248577 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5" Nov 02 15:30:04.259345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.258253 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerStarted","Data":"1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d"} Nov 02 15:30:04.262060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.261361 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerStarted","Data":"7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a"} Nov 02 15:30:04.265110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.264932 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerStarted","Data":"6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39"} Nov 02 15:30:04.270200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.269623 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerStarted","Data":"919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2"} Nov 02 15:30:04.272471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.272454 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn" Nov 02 15:30:04.277116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.277027 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerStarted","Data":"3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31"} Nov 02 15:30:04.580734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.580679 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" podUID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" containerName="payload-processing" containerID="cri-o://b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f" gracePeriod=25 Nov 02 15:30:04.709410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.705951 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" podUID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" containerName="payload-pre-processing" containerID="cri-o://baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e" gracePeriod=25 Nov 02 15:30:04.729674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.729612 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" podStartSLOduration=4.729588881 podStartE2EDuration="4.729588881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.706373151 +0000 UTC m=+2007.589975671" watchObservedRunningTime="2026-11-02 15:30:04.729588881 +0000 UTC m=+2007.613191403" Nov 02 15:30:04.847533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.845516 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" podStartSLOduration=4.845499629 podStartE2EDuration="4.845499629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.838395314 +0000 UTC m=+2007.721997834" watchObservedRunningTime="2026-11-02 15:30:04.845499629 +0000 UTC m=+2007.729102207" Nov 02 15:30:04.959054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:04.956569 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" podStartSLOduration=4.9565546430000005 podStartE2EDuration="4.956554643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.954687583 +0000 UTC m=+2007.838290103" watchObservedRunningTime="2026-11-02 15:30:04.956554643 +0000 UTC m=+2007.840157162" Nov 02 15:30:05.106326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.106299 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:30:05.114362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.114335 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\" (UniqueName: \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\") pod \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " Nov 02 15:30:05.114537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.114519 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\") pod \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\" (UID: \"e9e06e67-6bf0-4d82-8fcd-69a8170cf291\") " Nov 02 15:30:05.114854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.114833 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume" pod "e9e06e67-6bf0-4d82-8fcd-69a8170cf291" (UID: "e9e06e67-6bf0-4d82-8fcd-69a8170cf291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:05.115201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.115179 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:05.116539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.116520 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm" pod "e9e06e67-6bf0-4d82-8fcd-69a8170cf291" (UID: "e9e06e67-6bf0-4d82-8fcd-69a8170cf291"). InnerVolumeSpecName "kube-api-access-2x8xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:05.129687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.129224 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:30:05.133085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.133042 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" podStartSLOduration=5.133028163 podStartE2EDuration="5.133028163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:05.129363871 +0000 UTC m=+2008.012966389" watchObservedRunningTime="2026-11-02 15:30:05.133028163 +0000 UTC m=+2008.016630701" Nov 02 15:30:05.216030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.215997 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\") pod \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " Nov 02 15:30:05.216389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.216243 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\" (UniqueName: \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\") pod \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\" (UID: \"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85\") " Nov 02 15:30:05.216389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.216332 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume" pod "b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" (UID: "b6397750-8c5c-4ccf-bbf7-e88fbd23eb85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:05.216640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.216625 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8xm\" (UniqueName: \"kubernetes.io/projected/e9e06e67-6bf0-4d82-8fcd-69a8170cf291-kube-api-access-2x8xm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:05.216703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.216642 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:05.218408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.218388 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9" pod "b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" (UID: "b6397750-8c5c-4ccf-bbf7-e88fbd23eb85"). InnerVolumeSpecName "kube-api-access-7ldr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:05.245575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.245529 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" podStartSLOduration=5.245514743 podStartE2EDuration="5.245514743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:05.24427758 +0000 UTC m=+2008.127880102" watchObservedRunningTime="2026-11-02 15:30:05.245514743 +0000 UTC m=+2008.129117262" Nov 02 15:30:05.245764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.245739 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5692e221-7431-4fe8-9d4b-9a6afe036821" path="/var/lib/kubelet/pods/5692e221-7431-4fe8-9d4b-9a6afe036821/volumes" Nov 02 15:30:05.246409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.246378 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" path="/var/lib/kubelet/pods/b158bdd3-ecc6-460f-96b7-0753f3aa8c1d/volumes" Nov 02 15:30:05.281159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.281134 4469 generic.go:356] "Generic (PLEG): container finished" podID="d759e268-21dd-4530-9a59-780148841f93" containerID="76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3" exitCode=60 Nov 02 15:30:05.281275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.281190 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerDied","Data":"76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3"} Nov 02 15:30:05.281275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.281245 4469 scope.go:122] "RemoveContainer" containerID="abcf8e81d0d7c0b8f236fcea438c2916fc75a097e6313cfd2341f48e857eb5e4" Nov 02 15:30:05.281555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.281539 4469 scope.go:122] "RemoveContainer" containerID="76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3" Nov 02 15:30:05.281932 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.281903 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr_odh-ai-gateway-infra(d759e268-21dd-4530-9a59-780148841f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" podUID="d759e268-21dd-4530-9a59-780148841f93" Nov 02 15:30:05.282629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.282611 4469 generic.go:356] "Generic (PLEG): container finished" podID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" containerID="b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f" exitCode=0 Nov 02 15:30:05.282730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.282650 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" Nov 02 15:30:05.282730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.282694 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" event={"ID":"e9e06e67-6bf0-4d82-8fcd-69a8170cf291","Type":"ContainerDied","Data":"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f"} Nov 02 15:30:05.282898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.282757 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26" event={"ID":"e9e06e67-6bf0-4d82-8fcd-69a8170cf291","Type":"ContainerDied","Data":"2d96403b854245ee5df3e03c538841cc0f8e997d94f41c5b109e2a849183dc18"} Nov 02 15:30:05.284105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.284084 4469 generic.go:356] "Generic (PLEG): container finished" podID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" containerID="baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e" exitCode=0 Nov 02 15:30:05.284179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.284139 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" Nov 02 15:30:05.284179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.284169 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" event={"ID":"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85","Type":"ContainerDied","Data":"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e"} Nov 02 15:30:05.284264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.284213 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk" event={"ID":"b6397750-8c5c-4ccf-bbf7-e88fbd23eb85","Type":"ContainerDied","Data":"30e21f65852971e2ed2ee8c3523fe94d11c46df24c913e1a173670690f4a6ac7"} Nov 02 15:30:05.286361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.286338 4469 generic.go:356] "Generic (PLEG): container finished" podID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerID="f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc" exitCode=60 Nov 02 15:30:05.286532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.286423 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerDied","Data":"f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc"} Nov 02 15:30:05.286789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.286710 4469 scope.go:122] "RemoveContainer" containerID="f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc" Nov 02 15:30:05.287085 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.287065 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b_odh-ai-gateway-infra(65ba386c-d24f-480d-ad3b-5f66d742c4c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" Nov 02 15:30:05.288413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.288395 4469 generic.go:356] "Generic (PLEG): container finished" podID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerID="0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81" exitCode=60 Nov 02 15:30:05.288503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.288454 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" event={"ID":"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3","Type":"ContainerDied","Data":"0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81"} Nov 02 15:30:05.288686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.288672 4469 scope.go:122] "RemoveContainer" containerID="0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81" Nov 02 15:30:05.288925 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.288905 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279_odh-ai-gateway-infra(4b6b9172-a97d-49a0-a6ff-f4d126ad54d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" Nov 02 15:30:05.291464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.291442 4469 generic.go:356] "Generic (PLEG): container finished" podID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerID="1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d" exitCode=60 Nov 02 15:30:05.291547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.291522 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerDied","Data":"1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d"} Nov 02 15:30:05.291716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.291697 4469 scope.go:122] "RemoveContainer" containerID="1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d" Nov 02 15:30:05.291954 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.291932 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-6g76t_odh-ai-gateway-infra(c464c645-62d1-4a0c-a83e-83a51d1bf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" Nov 02 15:30:05.293505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.293485 4469 generic.go:356] "Generic (PLEG): container finished" podID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerID="7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a" exitCode=60 Nov 02 15:30:05.293593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.293554 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerDied","Data":"7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a"} Nov 02 15:30:05.293765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.293751 4469 scope.go:122] "RemoveContainer" containerID="7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a" Nov 02 15:30:05.294055 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.294030 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np_odh-ai-gateway-infra(2f58fc1b-bcc0-4a74-a2c3-95d224259fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" Nov 02 15:30:05.295444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.295417 4469 generic.go:356] "Generic (PLEG): container finished" podID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerID="6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39" exitCode=60 Nov 02 15:30:05.295520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.295440 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerDied","Data":"6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39"} Nov 02 15:30:05.295667 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.295652 4469 scope.go:122] "RemoveContainer" containerID="6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39" Nov 02 15:30:05.295917 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.295895 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6_odh-ai-gateway-infra(847b00d3-3f96-4521-936c-e47dde2fe6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" Nov 02 15:30:05.296745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.296723 4469 scope.go:122] "RemoveContainer" containerID="b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f" Nov 02 15:30:05.297486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.297468 4469 generic.go:356] "Generic (PLEG): container finished" podID="a298d4d4-4178-4ee1-995b-707345440cf4" containerID="919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2" exitCode=60 Nov 02 15:30:05.297569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.297529 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerDied","Data":"919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2"} Nov 02 15:30:05.297752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.297735 4469 scope.go:122] "RemoveContainer" containerID="919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2" Nov 02 15:30:05.297982 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.297959 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm_odh-ai-gateway-infra(a298d4d4-4178-4ee1-995b-707345440cf4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" Nov 02 15:30:05.299391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.299373 4469 generic.go:356] "Generic (PLEG): container finished" podID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerID="3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31" exitCode=60 Nov 02 15:30:05.299486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.299392 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerDied","Data":"3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31"} Nov 02 15:30:05.299674 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.299652 4469 scope.go:122] "RemoveContainer" containerID="3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31" Nov 02 15:30:05.299908 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.299889 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g_odh-ai-gateway-infra(b036f705-7118-4c89-82ab-92f321ebc7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" Nov 02 15:30:05.309820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.309774 4469 scope.go:122] "RemoveContainer" containerID="b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f" Nov 02 15:30:05.310438 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.310415 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f\": container with ID starting with b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f not found: ID does not exist" containerID="b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f" Nov 02 15:30:05.310546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.310449 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f"} err="failed to get container status \"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f\": rpc error: code = NotFound desc = could not find container \"b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f\": container with ID starting with b31fba1b391fb19525904ee09a520e6de89e71dde5b9d51a1d40be61787ab78f not found: ID does not exist" Nov 02 15:30:05.310546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.310469 4469 scope.go:122] "RemoveContainer" containerID="baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e" Nov 02 15:30:05.318384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.318360 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ldr9\" (UniqueName: \"kubernetes.io/projected/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85-kube-api-access-7ldr9\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:05.322988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.322968 4469 scope.go:122] "RemoveContainer" containerID="baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e" Nov 02 15:30:05.323275 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:05.323255 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e\": container with ID starting with baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e not found: ID does not exist" containerID="baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e" Nov 02 15:30:05.323367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.323285 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e"} err="failed to get container status \"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e\": rpc error: code = NotFound desc = could not find container \"baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e\": container with ID starting with baf86ef4bdcfd1ad774cc617f836e29921d9c51e6175f2ea11c030304496a62e not found: ID does not exist" Nov 02 15:30:05.323367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.323306 4469 scope.go:122] "RemoveContainer" containerID="20888799b22ef66b2ed37932174797b350370d8eb7ba42e4d7f1a2056460f978" Nov 02 15:30:05.339849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.339794 4469 scope.go:122] "RemoveContainer" containerID="facb5e751ad95b23e7e9b97593570a926d017b4b3aeb3f4f66ec64a018998d82" Nov 02 15:30:05.354828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.354793 4469 scope.go:122] "RemoveContainer" containerID="a7e9fd1dd2868324bacf5e51bfa4f24acdd9bf535759d03319c976435d378194" Nov 02 15:30:05.370161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.370067 4469 scope.go:122] "RemoveContainer" containerID="359807ce95371a867be1a6ced505c0b3eba3f210d87bfcec30535888c40ef377" Nov 02 15:30:05.384183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.384167 4469 scope.go:122] "RemoveContainer" containerID="33d1c32e08c0fb684961a6feeef2a44e7d05408786ca287d52bc47cdb4b3986e" Nov 02 15:30:05.396645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.396626 4469 scope.go:122] "RemoveContainer" containerID="fd4ab9297cbde264cbe1176eed280afc06189c907e5cb111742a457198189c76" Nov 02 15:30:05.409161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.409144 4469 scope.go:122] "RemoveContainer" containerID="16326ba086861a6149f17b3d7dfef2e1b80d7c4e91f9ece4a028a4a673a3e131" Nov 02 15:30:05.567888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.565562 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:05.591820 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.588229 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" podStartSLOduration=5.588206849 podStartE2EDuration="5.588206849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:05.540464526 +0000 UTC m=+2008.424067046" watchObservedRunningTime="2026-11-02 15:30:05.588206849 +0000 UTC m=+2008.471809369" Nov 02 15:30:05.783555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.783518 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:30:05.796143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:05.796115 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-w0-ba49d0-kserve-5dd6f49d59-wpwmn"] Nov 02 15:30:06.196599 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.196549 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:30:06.213390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.213359 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:30:06.224375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.224349 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b2f61-798968dcb7-2rkdk"] Nov 02 15:30:06.309519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.309493 4469 scope.go:122] "RemoveContainer" containerID="f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc" Nov 02 15:30:06.309896 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.309871 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b_odh-ai-gateway-infra(65ba386c-d24f-480d-ad3b-5f66d742c4c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" Nov 02 15:30:06.310985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.310963 4469 scope.go:122] "RemoveContainer" containerID="0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81" Nov 02 15:30:06.311269 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.311249 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279_odh-ai-gateway-infra(4b6b9172-a97d-49a0-a6ff-f4d126ad54d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" Nov 02 15:30:06.312477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.312456 4469 scope.go:122] "RemoveContainer" containerID="1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d" Nov 02 15:30:06.312752 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.312733 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-6g76t_odh-ai-gateway-infra(c464c645-62d1-4a0c-a83e-83a51d1bf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" Nov 02 15:30:06.313866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.313836 4469 scope.go:122] "RemoveContainer" containerID="7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a" Nov 02 15:30:06.314152 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.314129 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np_odh-ai-gateway-infra(2f58fc1b-bcc0-4a74-a2c3-95d224259fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" Nov 02 15:30:06.315310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.315292 4469 scope.go:122] "RemoveContainer" containerID="6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39" Nov 02 15:30:06.315546 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.315530 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6_odh-ai-gateway-infra(847b00d3-3f96-4521-936c-e47dde2fe6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" Nov 02 15:30:06.316737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.316717 4469 scope.go:122] "RemoveContainer" containerID="919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2" Nov 02 15:30:06.316945 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.316926 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm_odh-ai-gateway-infra(a298d4d4-4178-4ee1-995b-707345440cf4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" Nov 02 15:30:06.318167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.318150 4469 scope.go:122] "RemoveContainer" containerID="3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31" Nov 02 15:30:06.318363 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.318344 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g_odh-ai-gateway-infra(b036f705-7118-4c89-82ab-92f321ebc7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" Nov 02 15:30:06.319549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.319533 4469 scope.go:122] "RemoveContainer" containerID="76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3" Nov 02 15:30:06.319724 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:06.319707 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr_odh-ai-gateway-infra(d759e268-21dd-4530-9a59-780148841f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" podUID="d759e268-21dd-4530-9a59-780148841f93" Nov 02 15:30:06.623339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.623305 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:30:06.629724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.629692 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b2f61-78b8fcdb9c-ttp26"] Nov 02 15:30:06.972900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.972858 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:30:06.981908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:06.979825 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/premium-w0-ba49d0-kserve-595d9c564-794d5"] Nov 02 15:30:07.245637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:07.245540 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" path="/var/lib/kubelet/pods/1fa67505-1d76-438f-b61d-81b0bd53f704/volumes" Nov 02 15:30:07.246900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:07.246345 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" path="/var/lib/kubelet/pods/a2dbd84f-4471-454d-89d3-48f756e99e51/volumes" Nov 02 15:30:07.247549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:07.247524 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" path="/var/lib/kubelet/pods/b6397750-8c5c-4ccf-bbf7-e88fbd23eb85/volumes" Nov 02 15:30:07.248728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:07.248705 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" path="/var/lib/kubelet/pods/e9e06e67-6bf0-4d82-8fcd-69a8170cf291/volumes" Nov 02 15:30:08.233221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:08.233184 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:10.847662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:10.847622 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:10.876567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:10.876536 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:30:10.876833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:10.876785 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" containerID="cri-o://6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de" gracePeriod=30 Nov 02 15:30:14.141072 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:14.141027 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:14.141860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:14.141794 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:14.326217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:14.326178 4469 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:30:14.326405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:14.326252 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:30:15.233832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.233115 4469 scope.go:122] "RemoveContainer" containerID="6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39" Nov 02 15:30:15.236054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.236031 4469 scope.go:122] "RemoveContainer" containerID="3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31" Nov 02 15:30:15.271180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.271144 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:15.293877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.293849 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:15.603819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.602541 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:15.700734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.700695 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\") pod \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " Nov 02 15:30:15.701093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.701064 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\" (UniqueName: \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\") pod \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\" (UID: \"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3\") " Nov 02 15:30:15.701232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.701101 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca" pod "4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" (UID: "4b6b9172-a97d-49a0-a6ff-f4d126ad54d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:15.701747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.701724 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:15.703364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.703335 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm" pod "4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" (UID: "4b6b9172-a97d-49a0-a6ff-f4d126ad54d3"). InnerVolumeSpecName "kube-api-access-slxlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:15.803666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:15.803573 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxlm\" (UniqueName: \"kubernetes.io/projected/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3-kube-api-access-slxlm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.239902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.238317 4469 scope.go:122] "RemoveContainer" containerID="76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3" Nov 02 15:30:16.242232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.242212 4469 scope.go:122] "RemoveContainer" containerID="1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d" Nov 02 15:30:16.418764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.418727 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:30:16.468095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.467863 4469 generic.go:356] "Generic (PLEG): container finished" podID="7030d72e-a96f-420d-9d3f-a373b196c485" containerID="6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de" exitCode=0 Nov 02 15:30:16.468095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.467934 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" Nov 02 15:30:16.468095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.467953 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" event={"ID":"7030d72e-a96f-420d-9d3f-a373b196c485","Type":"ContainerDied","Data":"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de"} Nov 02 15:30:16.468095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.468030 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds" event={"ID":"7030d72e-a96f-420d-9d3f-a373b196c485","Type":"ContainerDied","Data":"e29b67267dfa0adb8ad2db1b62aaaa6464f8176aca14257afbad776356cf90de"} Nov 02 15:30:16.468095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.468066 4469 scope.go:122] "RemoveContainer" containerID="6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de" Nov 02 15:30:16.471881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.471309 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" event={"ID":"4b6b9172-a97d-49a0-a6ff-f4d126ad54d3","Type":"ContainerDied","Data":"754e9630d44fbe3b298b863b605e3d576359ffc326e265724b0b06e464f03c5f"} Nov 02 15:30:16.471881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.471410 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279" Nov 02 15:30:16.493809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.493698 4469 generic.go:356] "Generic (PLEG): container finished" podID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerID="ad57980c153f3efa7f1534482e664e7eb0d4df71a1e13e26c0da3bf15176eee9" exitCode=60 Nov 02 15:30:16.493809 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.493732 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerDied","Data":"ad57980c153f3efa7f1534482e664e7eb0d4df71a1e13e26c0da3bf15176eee9"} Nov 02 15:30:16.496266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.496245 4469 generic.go:356] "Generic (PLEG): container finished" podID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerID="6ab94a404a8fc9bcb05fd26d163f3fd7176a2a6137ffdbdec821f0e793d63fa4" exitCode=60 Nov 02 15:30:16.496393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.496293 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerDied","Data":"6ab94a404a8fc9bcb05fd26d163f3fd7176a2a6137ffdbdec821f0e793d63fa4"} Nov 02 15:30:16.500437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.500414 4469 scope.go:122] "RemoveContainer" containerID="6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:16.500793 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de\": container with ID starting with 6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de not found: ID does not exist" containerID="6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.500871 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de"} err="failed to get container status \"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de\": rpc error: code = NotFound desc = could not find container \"6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de\": container with ID starting with 6c97e3e3a745c299159d55520e0ef9c3b675b8eb1ec4713378a170c7696258de not found: ID does not exist" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.500902 4469 scope.go:122] "RemoveContainer" containerID="0607d5bc5b8befbf7abb2fe63708dcab910e92c3a965a31b8a10601010735b81" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.501506 4469 scope.go:122] "RemoveContainer" containerID="ad57980c153f3efa7f1534482e664e7eb0d4df71a1e13e26c0da3bf15176eee9" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:16.501946 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6_odh-ai-gateway-infra(847b00d3-3f96-4521-936c-e47dde2fe6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.502594 4469 scope.go:122] "RemoveContainer" containerID="6ab94a404a8fc9bcb05fd26d163f3fd7176a2a6137ffdbdec821f0e793d63fa4" Nov 02 15:30:16.503025 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:16.502977 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g_odh-ai-gateway-infra(b036f705-7118-4c89-82ab-92f321ebc7e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" Nov 02 15:30:16.515655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.515629 4469 scope.go:122] "RemoveContainer" containerID="6b0d2f7035093a0f7da49c1c8a64d3045ef7d25f04ace7f876e2f791ec034f39" Nov 02 15:30:16.531008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.530984 4469 scope.go:122] "RemoveContainer" containerID="3698fdb63f6e8121933820844e14af084eb07d80a1b51c3423298719068ada31" Nov 02 15:30:16.560508 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.560462 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.560677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.560584 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.560677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.560616 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.561230 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561203 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.561381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561270 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561514 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561860 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561884 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.561985 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562081 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\") pod \"7030d72e-a96f-420d-9d3f-a373b196c485\" (UID: \"7030d72e-a96f-420d-9d3f-a373b196c485\") " Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562149 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562308 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562867 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562891 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.562905 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.563556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.563493 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:16.566443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.566404 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:16.636378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.636016 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:16.651729 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.645023 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:30:16.652741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.652693 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:16.653122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.653092 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz" pod "7030d72e-a96f-420d-9d3f-a373b196c485" (UID: "7030d72e-a96f-420d-9d3f-a373b196c485"). InnerVolumeSpecName "kube-api-access-mptsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.668982 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptsz\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-kube-api-access-mptsz\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.669012 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7030d72e-a96f-420d-9d3f-a373b196c485-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.669027 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7030d72e-a96f-420d-9d3f-a373b196c485-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.669045 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.669059 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7030d72e-a96f-420d-9d3f-a373b196c485-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.689587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.669075 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7030d72e-a96f-420d-9d3f-a373b196c485-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.898493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.898459 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:16.955144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.955110 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:30:16.968973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.968938 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:16.974162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:16.974132 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eafc0d-29893890-fw279"] Nov 02 15:30:17.083432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.083299 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:30:17.087289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.087264 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-rnfds"] Nov 02 15:30:17.161487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.161448 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:17.248095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.248041 4469 scope.go:122] "RemoveContainer" containerID="f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc" Nov 02 15:30:17.257846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.257812 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" path="/var/lib/kubelet/pods/4b6b9172-a97d-49a0-a6ff-f4d126ad54d3/volumes" Nov 02 15:30:17.258376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.258361 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" path="/var/lib/kubelet/pods/7030d72e-a96f-420d-9d3f-a373b196c485/volumes" Nov 02 15:30:17.417491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.417401 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:17.507107 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.507075 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerStarted","Data":"e510471e6db25be4bd9a5b656a869274a346de08e7f4d90a1e958a27bd6822bd"} Nov 02 15:30:17.512976 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.512947 4469 generic.go:356] "Generic (PLEG): container finished" podID="d759e268-21dd-4530-9a59-780148841f93" containerID="18acca64d8cb7066705a5d9198bfc968930b08c8db420b5c9da216ecb15ddfa0" exitCode=60 Nov 02 15:30:17.513104 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.513020 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerDied","Data":"18acca64d8cb7066705a5d9198bfc968930b08c8db420b5c9da216ecb15ddfa0"} Nov 02 15:30:17.513104 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.513075 4469 scope.go:122] "RemoveContainer" containerID="76bf64b4479cda437b1953f6ce219f73339be9ad785b1ef19caf22916e804aa3" Nov 02 15:30:17.513410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.513389 4469 scope.go:122] "RemoveContainer" containerID="18acca64d8cb7066705a5d9198bfc968930b08c8db420b5c9da216ecb15ddfa0" Nov 02 15:30:17.513698 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:17.513675 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr_odh-ai-gateway-infra(d759e268-21dd-4530-9a59-780148841f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" podUID="d759e268-21dd-4530-9a59-780148841f93" Nov 02 15:30:17.675049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.674962 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:17.845610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.845581 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:17.869742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.869709 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:17.881920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:17.881891 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:18.010128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.010100 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\") pod \"847b00d3-3f96-4521-936c-e47dde2fe6e8\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " Nov 02 15:30:18.010278 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.010252 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\" (UniqueName: \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\") pod \"847b00d3-3f96-4521-936c-e47dde2fe6e8\" (UID: \"847b00d3-3f96-4521-936c-e47dde2fe6e8\") " Nov 02 15:30:18.010466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.010445 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca" pod "847b00d3-3f96-4521-936c-e47dde2fe6e8" (UID: "847b00d3-3f96-4521-936c-e47dde2fe6e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:18.010795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.010777 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847b00d3-3f96-4521-936c-e47dde2fe6e8-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:18.012453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.012431 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92" pod "847b00d3-3f96-4521-936c-e47dde2fe6e8" (UID: "847b00d3-3f96-4521-936c-e47dde2fe6e8"). InnerVolumeSpecName "kube-api-access-hqz92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:18.020400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.020380 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:18.131948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.119541 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqz92\" (UniqueName: \"kubernetes.io/projected/847b00d3-3f96-4521-936c-e47dde2fe6e8-kube-api-access-hqz92\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:18.234538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.234420 4469 scope.go:122] "RemoveContainer" containerID="919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2" Nov 02 15:30:18.239022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.237148 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\" (UniqueName: \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\") pod \"b036f705-7118-4c89-82ab-92f321ebc7e9\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " Nov 02 15:30:18.239022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.237232 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\") pod \"b036f705-7118-4c89-82ab-92f321ebc7e9\" (UID: \"b036f705-7118-4c89-82ab-92f321ebc7e9\") " Nov 02 15:30:18.240765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.239465 4469 scope.go:122] "RemoveContainer" containerID="7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a" Nov 02 15:30:18.241168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.241131 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca" pod "b036f705-7118-4c89-82ab-92f321ebc7e9" (UID: "b036f705-7118-4c89-82ab-92f321ebc7e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:18.248475 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.244995 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb" pod "b036f705-7118-4c89-82ab-92f321ebc7e9" (UID: "b036f705-7118-4c89-82ab-92f321ebc7e9"). InnerVolumeSpecName "kube-api-access-6xzhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:18.340039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.340009 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzhb\" (UniqueName: \"kubernetes.io/projected/b036f705-7118-4c89-82ab-92f321ebc7e9-kube-api-access-6xzhb\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:18.340216 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.340043 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b036f705-7118-4c89-82ab-92f321ebc7e9-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:18.521429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.521401 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" Nov 02 15:30:18.521618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.521423 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6" event={"ID":"847b00d3-3f96-4521-936c-e47dde2fe6e8","Type":"ContainerDied","Data":"611aa453c134033ccbf5a1d97fd0d99fdc8dae1a1d6a01ee8fb50e40aa58b07c"} Nov 02 15:30:18.521618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.521494 4469 scope.go:122] "RemoveContainer" containerID="ad57980c153f3efa7f1534482e664e7eb0d4df71a1e13e26c0da3bf15176eee9" Nov 02 15:30:18.522947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.522929 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" Nov 02 15:30:18.523069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.522953 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g" event={"ID":"b036f705-7118-4c89-82ab-92f321ebc7e9","Type":"ContainerDied","Data":"5696dd9fc504c305045be23f095776ab9bec6ac73a582ed2a048d4852571c0ee"} Nov 02 15:30:18.527443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.527420 4469 generic.go:356] "Generic (PLEG): container finished" podID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerID="198e8a0f74d55e881a2d075e99aba73c1c5433fc451d716762c6f33f3b8f16bb" exitCode=60 Nov 02 15:30:18.527564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.527486 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerDied","Data":"198e8a0f74d55e881a2d075e99aba73c1c5433fc451d716762c6f33f3b8f16bb"} Nov 02 15:30:18.527995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.527965 4469 scope.go:122] "RemoveContainer" containerID="198e8a0f74d55e881a2d075e99aba73c1c5433fc451d716762c6f33f3b8f16bb" Nov 02 15:30:18.528310 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:18.528288 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b_odh-ai-gateway-infra(65ba386c-d24f-480d-ad3b-5f66d742c4c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" Nov 02 15:30:18.530952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.530929 4469 generic.go:356] "Generic (PLEG): container finished" podID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerID="e510471e6db25be4bd9a5b656a869274a346de08e7f4d90a1e958a27bd6822bd" exitCode=60 Nov 02 15:30:18.531064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.530994 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerDied","Data":"e510471e6db25be4bd9a5b656a869274a346de08e7f4d90a1e958a27bd6822bd"} Nov 02 15:30:18.531384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.531367 4469 scope.go:122] "RemoveContainer" containerID="e510471e6db25be4bd9a5b656a869274a346de08e7f4d90a1e958a27bd6822bd" Nov 02 15:30:18.531638 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:18.531619 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-6g76t_odh-ai-gateway-infra(c464c645-62d1-4a0c-a83e-83a51d1bf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" Nov 02 15:30:18.535573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.535551 4469 scope.go:122] "RemoveContainer" containerID="6ab94a404a8fc9bcb05fd26d163f3fd7176a2a6137ffdbdec821f0e793d63fa4" Nov 02 15:30:18.544936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.544914 4469 scope.go:122] "RemoveContainer" containerID="f9b817ef25bcbde5cb55a96615d3fd75cb7b4bc8edb66631f166afd7b94c5bdc" Nov 02 15:30:18.557778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.557763 4469 scope.go:122] "RemoveContainer" containerID="1ea7599929b6d2eeaa971906c30f861db76c348a0959608ab8fd80ca9c010a5d" Nov 02 15:30:18.671751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.671718 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:18.696018 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.695979 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:18.702225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.702201 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893890-d7pv6"] Nov 02 15:30:18.794602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.794514 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:18.958731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.958703 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:18.966022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.965999 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba49d0-29893890-zmx6g"] Nov 02 15:30:18.971435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:18.971414 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:19.050150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.050068 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\" (UniqueName: \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\") pod \"d759e268-21dd-4530-9a59-780148841f93\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " Nov 02 15:30:19.050325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.050175 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\") pod \"d759e268-21dd-4530-9a59-780148841f93\" (UID: \"d759e268-21dd-4530-9a59-780148841f93\") " Nov 02 15:30:19.050509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.050487 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca" pod "d759e268-21dd-4530-9a59-780148841f93" (UID: "d759e268-21dd-4530-9a59-780148841f93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:19.050670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.050651 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d759e268-21dd-4530-9a59-780148841f93-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.052238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.052213 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f" pod "d759e268-21dd-4530-9a59-780148841f93" (UID: "d759e268-21dd-4530-9a59-780148841f93"). InnerVolumeSpecName "kube-api-access-khh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:19.151646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.151615 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khh2f\" (UniqueName: \"kubernetes.io/projected/d759e268-21dd-4530-9a59-780148841f93-kube-api-access-khh2f\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.245844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.245789 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" path="/var/lib/kubelet/pods/847b00d3-3f96-4521-936c-e47dde2fe6e8/volumes" Nov 02 15:30:19.246509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.246485 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" path="/var/lib/kubelet/pods/b036f705-7118-4c89-82ab-92f321ebc7e9/volumes" Nov 02 15:30:19.543617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.543590 4469 generic.go:356] "Generic (PLEG): container finished" podID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerID="3a25cf7f89ff4e46677e6071c8bc879d91a3d261e0372c7f31fc702ea306a20f" exitCode=60 Nov 02 15:30:19.544016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.543665 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerDied","Data":"3a25cf7f89ff4e46677e6071c8bc879d91a3d261e0372c7f31fc702ea306a20f"} Nov 02 15:30:19.544016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.543739 4469 scope.go:122] "RemoveContainer" containerID="7d44019e612da6f254807ca8168ff19fe34e2e01ba24270be4d712b0f1345d4a" Nov 02 15:30:19.544390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.544233 4469 scope.go:122] "RemoveContainer" containerID="3a25cf7f89ff4e46677e6071c8bc879d91a3d261e0372c7f31fc702ea306a20f" Nov 02 15:30:19.544639 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:19.544616 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np_odh-ai-gateway-infra(2f58fc1b-bcc0-4a74-a2c3-95d224259fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" Nov 02 15:30:19.546834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.546794 4469 generic.go:356] "Generic (PLEG): container finished" podID="a298d4d4-4178-4ee1-995b-707345440cf4" containerID="f43ba2b3056e1bc3558fd8cd17181112065dacc5439438a99525599c075aee99" exitCode=60 Nov 02 15:30:19.546834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.546826 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerDied","Data":"f43ba2b3056e1bc3558fd8cd17181112065dacc5439438a99525599c075aee99"} Nov 02 15:30:19.547244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.547225 4469 scope.go:122] "RemoveContainer" containerID="f43ba2b3056e1bc3558fd8cd17181112065dacc5439438a99525599c075aee99" Nov 02 15:30:19.547597 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:19.547569 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm_odh-ai-gateway-infra(a298d4d4-4178-4ee1-995b-707345440cf4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" Nov 02 15:30:19.549145 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.549122 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" event={"ID":"d759e268-21dd-4530-9a59-780148841f93","Type":"ContainerDied","Data":"15e9af30c2bdc834fbfbcff19234922928a4e2d9f32a9627b820ed841fca0b47"} Nov 02 15:30:19.549267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.549150 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr" Nov 02 15:30:19.584023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.584002 4469 scope.go:122] "RemoveContainer" containerID="919bd5a31f72e475a5be85f442aa9f3b7fbc06d3346d9aa60a14d9b960a7ecf2" Nov 02 15:30:19.707935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.707567 4469 scope.go:122] "RemoveContainer" containerID="18acca64d8cb7066705a5d9198bfc968930b08c8db420b5c9da216ecb15ddfa0" Nov 02 15:30:19.744844 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.742108 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:19.759857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.758217 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:19.802958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.802910 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:30:19.839231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.839148 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:19.861306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.861218 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\") pod \"44660d8c-1252-4503-8303-afffee9fb66d\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " Nov 02 15:30:19.861306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.861272 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\" (UniqueName: \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\") pod \"c464c645-62d1-4a0c-a83e-83a51d1bf408\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.861347 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\" (UniqueName: \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\") pod \"44660d8c-1252-4503-8303-afffee9fb66d\" (UID: \"44660d8c-1252-4503-8303-afffee9fb66d\") " Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.861649 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\") pod \"c464c645-62d1-4a0c-a83e-83a51d1bf408\" (UID: \"c464c645-62d1-4a0c-a83e-83a51d1bf408\") " Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.862247 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume" pod "44660d8c-1252-4503-8303-afffee9fb66d" (UID: "44660d8c-1252-4503-8303-afffee9fb66d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.862409 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44660d8c-1252-4503-8303-afffee9fb66d-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.863056 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca" pod "c464c645-62d1-4a0c-a83e-83a51d1bf408" (UID: "c464c645-62d1-4a0c-a83e-83a51d1bf408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:19.865087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.864128 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct" pod "c464c645-62d1-4a0c-a83e-83a51d1bf408" (UID: "c464c645-62d1-4a0c-a83e-83a51d1bf408"). InnerVolumeSpecName "kube-api-access-smlct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:19.866214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.866189 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw" pod "44660d8c-1252-4503-8303-afffee9fb66d" (UID: "44660d8c-1252-4503-8303-afffee9fb66d"). InnerVolumeSpecName "kube-api-access-vzckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:19.909105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.908030 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:19.910317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.910255 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:30:19.915882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.915750 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893890-26jpr"] Nov 02 15:30:19.962952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.962927 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\") pod \"6bf67fec-d305-44f3-ad86-73424427b88e\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " Nov 02 15:30:19.963112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963071 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\" (UniqueName: \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\") pod \"6bf67fec-d305-44f3-ad86-73424427b88e\" (UID: \"6bf67fec-d305-44f3-ad86-73424427b88e\") " Nov 02 15:30:19.963351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963329 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume" pod "6bf67fec-d305-44f3-ad86-73424427b88e" (UID: "6bf67fec-d305-44f3-ad86-73424427b88e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:19.963838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963812 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzckw\" (UniqueName: \"kubernetes.io/projected/44660d8c-1252-4503-8303-afffee9fb66d-kube-api-access-vzckw\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.963928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963842 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c464c645-62d1-4a0c-a83e-83a51d1bf408-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.963928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963855 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlct\" (UniqueName: \"kubernetes.io/projected/c464c645-62d1-4a0c-a83e-83a51d1bf408-kube-api-access-smlct\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.963928 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.963864 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf67fec-d305-44f3-ad86-73424427b88e-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.965764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.965740 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct" pod "6bf67fec-d305-44f3-ad86-73424427b88e" (UID: "6bf67fec-d305-44f3-ad86-73424427b88e"). InnerVolumeSpecName "kube-api-access-9w9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:19.968253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:19.968221 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:20.031213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.031186 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:20.064832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.064776 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\" (UniqueName: \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\") pod \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " Nov 02 15:30:20.064972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.064939 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\") pod \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\" (UID: \"65ba386c-d24f-480d-ad3b-5f66d742c4c4\") " Nov 02 15:30:20.065305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.065280 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca" pod "65ba386c-d24f-480d-ad3b-5f66d742c4c4" (UID: "65ba386c-d24f-480d-ad3b-5f66d742c4c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:20.065380 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.065324 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9ct\" (UniqueName: \"kubernetes.io/projected/6bf67fec-d305-44f3-ad86-73424427b88e-kube-api-access-9w9ct\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:20.066875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.066850 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn" pod "65ba386c-d24f-480d-ad3b-5f66d742c4c4" (UID: "65ba386c-d24f-480d-ad3b-5f66d742c4c4"). InnerVolumeSpecName "kube-api-access-67kbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:20.166766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.166680 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba386c-d24f-480d-ad3b-5f66d742c4c4-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:20.166766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.166711 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kbn\" (UniqueName: \"kubernetes.io/projected/65ba386c-d24f-480d-ad3b-5f66d742c4c4-kube-api-access-67kbn\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:20.569267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.569237 4469 generic.go:356] "Generic (PLEG): container finished" podID="44660d8c-1252-4503-8303-afffee9fb66d" containerID="c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b" exitCode=0 Nov 02 15:30:20.569776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.569286 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" Nov 02 15:30:20.569776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.569336 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" event={"ID":"44660d8c-1252-4503-8303-afffee9fb66d","Type":"ContainerDied","Data":"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b"} Nov 02 15:30:20.569776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.569388 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh" event={"ID":"44660d8c-1252-4503-8303-afffee9fb66d","Type":"ContainerDied","Data":"74689d21a181e53c0e61c7b3764f5bc715e610e001a6a813188ed498b0c3d64e"} Nov 02 15:30:20.569776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.569420 4469 scope.go:122] "RemoveContainer" containerID="c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b" Nov 02 15:30:20.575837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.571126 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" Nov 02 15:30:20.575837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.571152 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b" event={"ID":"65ba386c-d24f-480d-ad3b-5f66d742c4c4","Type":"ContainerDied","Data":"6bace28caa424c12e71f3200873952f16394c2353de1e9b4f4e029f9726d4c92"} Nov 02 15:30:20.575837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.572760 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" Nov 02 15:30:20.575837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.572778 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t" event={"ID":"c464c645-62d1-4a0c-a83e-83a51d1bf408","Type":"ContainerDied","Data":"4ea9dd0e160b0bd5239b42158e985c28dd052efd3fb65e1b76c7a48cd7b57d99"} Nov 02 15:30:20.578741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.578716 4469 generic.go:356] "Generic (PLEG): container finished" podID="6bf67fec-d305-44f3-ad86-73424427b88e" containerID="cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4" exitCode=0 Nov 02 15:30:20.578877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.578751 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" Nov 02 15:30:20.578975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.578878 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" event={"ID":"6bf67fec-d305-44f3-ad86-73424427b88e","Type":"ContainerDied","Data":"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4"} Nov 02 15:30:20.578975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.578916 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf" event={"ID":"6bf67fec-d305-44f3-ad86-73424427b88e","Type":"ContainerDied","Data":"355874bb971ba4769491113d8cddf7d14d122a4a10444602dfdd824128f5c2f8"} Nov 02 15:30:20.581895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.581877 4469 scope.go:122] "RemoveContainer" containerID="c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b" Nov 02 15:30:20.582417 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:20.582383 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b\": container with ID starting with c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b not found: ID does not exist" containerID="c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b" Nov 02 15:30:20.582524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.582428 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b"} err="failed to get container status \"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b\": rpc error: code = NotFound desc = could not find container \"c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b\": container with ID starting with c8bdbddf6d4e11c7eba9242f46aae8518e53817ad060cf7b7a6840153635bf6b not found: ID does not exist" Nov 02 15:30:20.582524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.582452 4469 scope.go:122] "RemoveContainer" containerID="198e8a0f74d55e881a2d075e99aba73c1c5433fc451d716762c6f33f3b8f16bb" Nov 02 15:30:20.592386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.592367 4469 scope.go:122] "RemoveContainer" containerID="e510471e6db25be4bd9a5b656a869274a346de08e7f4d90a1e958a27bd6822bd" Nov 02 15:30:20.600955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.600936 4469 scope.go:122] "RemoveContainer" containerID="cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4" Nov 02 15:30:20.609236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.609217 4469 scope.go:122] "RemoveContainer" containerID="cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4" Nov 02 15:30:20.609506 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:20.609487 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4\": container with ID starting with cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4 not found: ID does not exist" containerID="cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4" Nov 02 15:30:20.609566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.609512 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4"} err="failed to get container status \"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4\": rpc error: code = NotFound desc = could not find container \"cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4\": container with ID starting with cc8b777e042f7d5027e8b0e6a5735b6f200a7077322fc6d1a8273fcfcb2e71c4 not found: ID does not exist" Nov 02 15:30:20.731751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.731713 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:20.735180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.735152 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-6g76t"] Nov 02 15:30:20.804414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.804377 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:20.843087 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.842996 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:30:20.851701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.851669 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ba49d0-79cdb5449f-m7srf"] Nov 02 15:30:20.962451 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.962423 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:20.981071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.981044 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-de9c4ba4-29893890-hnt2b"] Nov 02 15:30:20.991514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:20.991489 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:21.041937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.041912 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:21.078899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.078870 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\" (UniqueName: \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\") pod \"a298d4d4-4178-4ee1-995b-707345440cf4\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " Nov 02 15:30:21.079058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.078997 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\") pod \"a298d4d4-4178-4ee1-995b-707345440cf4\" (UID: \"a298d4d4-4178-4ee1-995b-707345440cf4\") " Nov 02 15:30:21.079395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.079367 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca" pod "a298d4d4-4178-4ee1-995b-707345440cf4" (UID: "a298d4d4-4178-4ee1-995b-707345440cf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.079717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.079687 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a298d4d4-4178-4ee1-995b-707345440cf4-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.081002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.080978 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f" pod "a298d4d4-4178-4ee1-995b-707345440cf4" (UID: "a298d4d4-4178-4ee1-995b-707345440cf4"). InnerVolumeSpecName "kube-api-access-fdc7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.086769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.086747 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:30:21.092387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.092365 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ba49d0-c49b65df5-qvfjh"] Nov 02 15:30:21.181187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.181160 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc7f\" (UniqueName: \"kubernetes.io/projected/a298d4d4-4178-4ee1-995b-707345440cf4-kube-api-access-fdc7f\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.203188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.203162 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:21.245261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.245223 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44660d8c-1252-4503-8303-afffee9fb66d" path="/var/lib/kubelet/pods/44660d8c-1252-4503-8303-afffee9fb66d/volumes" Nov 02 15:30:21.245956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.245931 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" path="/var/lib/kubelet/pods/65ba386c-d24f-480d-ad3b-5f66d742c4c4/volumes" Nov 02 15:30:21.246654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.246513 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" path="/var/lib/kubelet/pods/6bf67fec-d305-44f3-ad86-73424427b88e/volumes" Nov 02 15:30:21.247621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.247521 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" path="/var/lib/kubelet/pods/c464c645-62d1-4a0c-a83e-83a51d1bf408/volumes" Nov 02 15:30:21.248124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.248103 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d759e268-21dd-4530-9a59-780148841f93" path="/var/lib/kubelet/pods/d759e268-21dd-4530-9a59-780148841f93/volumes" Nov 02 15:30:21.281559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.281534 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\") pod \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " Nov 02 15:30:21.281680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.281667 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\" (UniqueName: \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\") pod \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\" (UID: \"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd\") " Nov 02 15:30:21.281920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.281898 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca" pod "2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" (UID: "2f58fc1b-bcc0-4a74-a2c3-95d224259fcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.282128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.282112 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.283703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.283678 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8" pod "2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" (UID: "2f58fc1b-bcc0-4a74-a2c3-95d224259fcd"). InnerVolumeSpecName "kube-api-access-d6hc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.384108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.384034 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6hc8\" (UniqueName: \"kubernetes.io/projected/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd-kube-api-access-d6hc8\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.609560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.609528 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" Nov 02 15:30:21.609972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.609531 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np" event={"ID":"2f58fc1b-bcc0-4a74-a2c3-95d224259fcd","Type":"ContainerDied","Data":"58392d213bafe271e6c2446faf1d082eb873fe16ede7954a14bb876941246a8c"} Nov 02 15:30:21.609972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.609701 4469 scope.go:122] "RemoveContainer" containerID="3a25cf7f89ff4e46677e6071c8bc879d91a3d261e0372c7f31fc702ea306a20f" Nov 02 15:30:21.613722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.613690 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" event={"ID":"a298d4d4-4178-4ee1-995b-707345440cf4","Type":"ContainerDied","Data":"ea7218a177099c218ab42d843839a57ffba339c7c193c51fb4a8a7343edd2334"} Nov 02 15:30:21.613885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.613699 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm" Nov 02 15:30:21.622916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.622897 4469 scope.go:122] "RemoveContainer" containerID="f43ba2b3056e1bc3558fd8cd17181112065dacc5439438a99525599c075aee99" Nov 02 15:30:21.785535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.785500 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:21.789140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.789110 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893890-p6vxm"] Nov 02 15:30:21.895664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.895584 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:21.904781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:21.904752 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d6b9c6-29893890-fc5np"] Nov 02 15:30:22.500955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:22.500100 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:23.240414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:23.240381 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" path="/var/lib/kubelet/pods/2f58fc1b-bcc0-4a74-a2c3-95d224259fcd/volumes" Nov 02 15:30:23.240846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:23.240829 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" path="/var/lib/kubelet/pods/a298d4d4-4178-4ee1-995b-707345440cf4/volumes" Nov 02 15:30:25.507840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:25.506504 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:25.713924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:25.713873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:25.947878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:25.942266 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:26.943481 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:26.943439 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:27.700500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:27.700446 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:27.903842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:27.903812 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:32.625940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:32.625908 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:30:32.765669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:32.765635 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:33.189030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:33.188369 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:33.397876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:33.397845 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:36.186885 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:30:36.186848 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:30:38.116439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:38.116405 4469 scope.go:122] "RemoveContainer" containerID="cd28e19ba6ee284563ae16e43c18e2652b9637e276f091d22afb07bc6b1e8ad8" Nov 02 15:30:38.140334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:38.138234 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:38.340410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:38.340384 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:40.305432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:40.305394 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:30:40.659920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:40.658217 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:40.866108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:40.866067 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:43.625598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:43.625560 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.163281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:44.163151 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.501825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:44.500110 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.705892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:44.705856 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.913054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:44.912972 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:45.128472 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:45.128438 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:45.306253 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:45.306214 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" containerID="cri-o://5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d" gracePeriod=25 Nov 02 15:30:47.647959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:47.647912 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" containerID="cri-o://80d5b8d1bddcb2dabee1ae205d04842bd794ad9b14506455554e09be2bc59da9" gracePeriod=45 Nov 02 15:30:47.890640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:47.890604 4469 generic.go:356] "Generic (PLEG): container finished" podID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerID="80d5b8d1bddcb2dabee1ae205d04842bd794ad9b14506455554e09be2bc59da9" exitCode=0 Nov 02 15:30:47.890877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:47.890685 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" event={"ID":"7697f54a-9b62-46cf-aa9d-aa753630d01b","Type":"ContainerDied","Data":"80d5b8d1bddcb2dabee1ae205d04842bd794ad9b14506455554e09be2bc59da9"} Nov 02 15:30:47.932708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:47.932685 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:30:48.051895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.051857 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.052114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.051905 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\" (UniqueName: \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.052114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.052043 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.052114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.052096 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.052286 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.052162 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.054410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.052383 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\") pod \"7697f54a-9b62-46cf-aa9d-aa753630d01b\" (UID: \"7697f54a-9b62-46cf-aa9d-aa753630d01b\") " Nov 02 15:30:48.054410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.053427 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:48.054410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.053912 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:48.054410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.054328 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.054410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.054353 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.055034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.054486 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:48.057236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.057197 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:30:48.058694 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.058664 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:48.058879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.058854 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv" pod "7697f54a-9b62-46cf-aa9d-aa753630d01b" (UID: "7697f54a-9b62-46cf-aa9d-aa753630d01b"). InnerVolumeSpecName "kube-api-access-k8lvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:48.155592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.155506 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.155592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.155538 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7697f54a-9b62-46cf-aa9d-aa753630d01b-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.155592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.155552 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7697f54a-9b62-46cf-aa9d-aa753630d01b-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.155592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.155566 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lvv\" (UniqueName: \"kubernetes.io/projected/7697f54a-9b62-46cf-aa9d-aa753630d01b-kube-api-access-k8lvv\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:30:48.883719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.883673 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:30:48.898979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.898936 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" event={"ID":"7697f54a-9b62-46cf-aa9d-aa753630d01b","Type":"ContainerDied","Data":"b5710ed01094b46aee06ab2f7d42fc00a991c1763108785c3541f68948d5c13c"} Nov 02 15:30:48.899130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.899008 4469 scope.go:122] "RemoveContainer" containerID="80d5b8d1bddcb2dabee1ae205d04842bd794ad9b14506455554e09be2bc59da9" Nov 02 15:30:48.899255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:48.899231 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt" Nov 02 15:30:49.064071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:49.064033 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:30:49.070361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:49.070330 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/premium-w4-6dc69b-kserve-5746d87898-48rtt"] Nov 02 15:30:49.239517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:49.239485 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" path="/var/lib/kubelet/pods/7697f54a-9b62-46cf-aa9d-aa753630d01b/volumes" Nov 02 15:30:49.258545 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:49.258509 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:49.847188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:49.847148 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.250755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:50.250696 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.498445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:50.498415 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:51.533715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:51.533674 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:51.746097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:51.746055 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:51.968991 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:51.963503 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:53.944637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:53.944602 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:54.702845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:54.700889 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:54.903680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:54.903645 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:57.145436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:57.145392 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:57.355581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:57.355540 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:58.832967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:58.832791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:59.040123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:59.040074 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:59.257542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:59.257503 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:30:59.927916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:30:59.927866 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:02.101687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:02.101649 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:02.318205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:02.318158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:05.432458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.431016 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:05.637842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.632077 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:31:05.706909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.706877 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\") pod \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " Nov 02 15:31:05.706909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.706914 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\" (UniqueName: \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\") pod \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\" (UID: \"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9\") " Nov 02 15:31:05.707340 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.707309 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume" pod "4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" (UID: "4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:05.709074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.709052 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c" pod "4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" (UID: "4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9"). InnerVolumeSpecName "kube-api-access-rr72c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:05.808993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.808956 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.808993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:05.808987 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr72c\" (UniqueName: \"kubernetes.io/projected/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9-kube-api-access-rr72c\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:06.067400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.067363 4469 generic.go:356] "Generic (PLEG): container finished" podID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerID="5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d" exitCode=0 Nov 02 15:31:06.067584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.067409 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" Nov 02 15:31:06.067584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.067444 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" event={"ID":"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9","Type":"ContainerDied","Data":"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d"} Nov 02 15:31:06.067584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.067498 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4" event={"ID":"4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9","Type":"ContainerDied","Data":"06715cfd7a7af83691bf8f3e09b71e366e8e4c182caec5adf817f072401adb11"} Nov 02 15:31:06.067584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.067522 4469 scope.go:122] "RemoveContainer" containerID="5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d" Nov 02 15:31:06.080582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.080561 4469 scope.go:122] "RemoveContainer" containerID="5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d" Nov 02 15:31:06.080959 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:06.080939 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d\": container with ID starting with 5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d not found: ID does not exist" containerID="5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d" Nov 02 15:31:06.081003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.080968 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d"} err="failed to get container status \"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d\": rpc error: code = NotFound desc = could not find container \"5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d\": container with ID starting with 5362595addce0dd8e1a68acb7e0daf332ccddcc711d96cf75d66fa11e8ddc39d not found: ID does not exist" Nov 02 15:31:06.187143 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:06.187106 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:31:06.233880 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.233852 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:31:06.282116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:06.282077 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6dc69b-68d55b7995-kl9q4"] Nov 02 15:31:07.238957 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:07.238923 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" path="/var/lib/kubelet/pods/4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9/volumes" Nov 02 15:31:08.564045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:08.564007 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:09.786640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:09.786593 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.080662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:11.080623 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.293048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:11.292997 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.501715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:11.500133 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.712037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:11.712000 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.930846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:11.930714 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:12.400116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:12.400041 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:12.847759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:12.847725 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:13.238523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:13.238483 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:13.445675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:13.445647 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:13.883708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:13.883674 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:17.957384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:17.957347 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:18.733556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:18.733505 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:18.957745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:18.957703 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:19.137984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:19.137903 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:31:19.138132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:19.138107 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" containerID="cri-o://448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04" gracePeriod=30 Nov 02 15:31:19.193395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:19.193357 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:20.745822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:20.745530 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:20.954502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:20.954464 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:21.045047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:21.044940 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:31:21.045555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:21.045509 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" containerID="cri-o://2200a7ff56e036f289c6d8d708399d3eba24fa2fe440dc1bdcef143a90bc1f92" gracePeriod=30 Nov 02 15:31:21.203449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:21.203413 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:22.070062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:22.070026 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:31:22.070476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:22.070095 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:31:22.434120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:22.434046 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:31:22.434120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:22.434105 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:31:24.044843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.044758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:24.421074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.421050 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:31:24.539878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.539837 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.539895 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.539964 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540043 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540118 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540146 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540259 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:24.540435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540300 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540365 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540411 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\") pod \"11abbca7-0516-43c8-9547-954453117952\" (UID: \"11abbca7-0516-43c8-9547-954453117952\") " Nov 02 15:31:24.540593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540532 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:24.540593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.540547 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:24.541073 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541003 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:24.541186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541109 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:24.541460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541378 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.541460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541402 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.541460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541416 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11abbca7-0516-43c8-9547-954453117952-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.541460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541429 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.541460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.541442 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.542661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.542626 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:24.542768 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.542715 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "kube-api-access-n296r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:24.542856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.542790 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:24.542954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.542935 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token" pod "11abbca7-0516-43c8-9547-954453117952" (UID: "11abbca7-0516-43c8-9547-954453117952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:24.643057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.642983 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.643057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.643011 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11abbca7-0516-43c8-9547-954453117952-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.643057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.643022 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n296r\" (UniqueName: \"kubernetes.io/projected/11abbca7-0516-43c8-9547-954453117952-kube-api-access-n296r\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:24.643057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:24.643034 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11abbca7-0516-43c8-9547-954453117952-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:25.245684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.245647 4469 generic.go:356] "Generic (PLEG): container finished" podID="11abbca7-0516-43c8-9547-954453117952" containerID="448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04" exitCode=0 Nov 02 15:31:25.246191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.245920 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerDied","Data":"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04"} Nov 02 15:31:25.246191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.245970 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" event={"ID":"11abbca7-0516-43c8-9547-954453117952","Type":"ContainerDied","Data":"50a3c6330f8053d85267869f726d9e1ce504aeb6052fb86d45424f8bb4fcb8c8"} Nov 02 15:31:25.246191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.246000 4469 scope.go:122] "RemoveContainer" containerID="448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04" Nov 02 15:31:25.246375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.246198 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh" Nov 02 15:31:25.261709 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.261416 4469 scope.go:122] "RemoveContainer" containerID="ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a" Nov 02 15:31:25.270624 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.270602 4469 scope.go:122] "RemoveContainer" containerID="448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04" Nov 02 15:31:25.270978 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:25.270947 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04\": container with ID starting with 448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04 not found: ID does not exist" containerID="448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04" Nov 02 15:31:25.271084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.270989 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04"} err="failed to get container status \"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04\": rpc error: code = NotFound desc = could not find container \"448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04\": container with ID starting with 448b711c76e77e921c18ade2af53c7090830986fd07433d2afcc0a9f51be1a04 not found: ID does not exist" Nov 02 15:31:25.271084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.271012 4469 scope.go:122] "RemoveContainer" containerID="ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a" Nov 02 15:31:25.271313 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:25.271290 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a\": container with ID starting with ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a not found: ID does not exist" containerID="ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a" Nov 02 15:31:25.271383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.271322 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a"} err="failed to get container status \"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a\": rpc error: code = NotFound desc = could not find container \"ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a\": container with ID starting with ef221d2d48f752e08c95d0f1533b8883dccd9c386f289dfcc905fbe03ce9898a not found: ID does not exist" Nov 02 15:31:25.415394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.415359 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:31:25.466646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:25.466613 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-559973-openshift-default-5974f778dc-frvnh"] Nov 02 15:31:26.263732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.258691 4469 generic.go:356] "Generic (PLEG): container finished" podID="e7defa54-df39-4771-b93b-209602778580" containerID="2200a7ff56e036f289c6d8d708399d3eba24fa2fe440dc1bdcef143a90bc1f92" exitCode=0 Nov 02 15:31:26.263732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.258814 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerDied","Data":"2200a7ff56e036f289c6d8d708399d3eba24fa2fe440dc1bdcef143a90bc1f92"} Nov 02 15:31:26.263732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.258876 4469 scope.go:122] "RemoveContainer" containerID="1f8a1060b342e1c335e2602bd1a0b3ce0ed0837ac4dd560430408c2b1156bfe2" Nov 02 15:31:26.389135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.389112 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:31:26.471731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.471703 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.471930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.471747 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.471930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.471778 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.471930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.471912 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.471990 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472056 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:26.472117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472103 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472117 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:26.472268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472180 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472270 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472410 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:26.472551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472459 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\") pod \"e7defa54-df39-4771-b93b-209602778580\" (UID: \"e7defa54-df39-4771-b93b-209602778580\") " Nov 02 15:31:26.472708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472687 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:26.472810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.472723 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:26.473286 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.473262 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.473419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.473290 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defa54-df39-4771-b93b-209602778580-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.473419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.473305 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.473419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.473318 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.473419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.473332 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.474407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.474383 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:26.474642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.474619 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:26.474869 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.474850 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "kube-api-access-qck7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:26.474950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.474932 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo" pod "e7defa54-df39-4771-b93b-209602778580" (UID: "e7defa54-df39-4771-b93b-209602778580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:26.574967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.574934 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defa54-df39-4771-b93b-209602778580-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.574967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.574964 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qck7d\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-kube-api-access-qck7d\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.574967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.574974 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defa54-df39-4771-b93b-209602778580-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:26.575210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:26.574985 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defa54-df39-4771-b93b-209602778580-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:27.241090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.241058 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11abbca7-0516-43c8-9547-954453117952" path="/var/lib/kubelet/pods/11abbca7-0516-43c8-9547-954453117952/volumes" Nov 02 15:31:27.267584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.267558 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" Nov 02 15:31:27.268002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.267604 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2" event={"ID":"e7defa54-df39-4771-b93b-209602778580","Type":"ContainerDied","Data":"5a8e6c4f61b054f382702eaed2a08bc55c5939ace5e4d5b4193fbe555a76aa97"} Nov 02 15:31:27.268002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.267676 4469 scope.go:122] "RemoveContainer" containerID="2200a7ff56e036f289c6d8d708399d3eba24fa2fe440dc1bdcef143a90bc1f92" Nov 02 15:31:27.484236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.484204 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:31:27.534463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:27.534383 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b2f61-openshift-default-6466884b9f-5pxf2"] Nov 02 15:31:29.156677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:29.156531 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:29.240496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:29.240463 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7defa54-df39-4771-b93b-209602778580" path="/var/lib/kubelet/pods/e7defa54-df39-4771-b93b-209602778580/volumes" Nov 02 15:31:31.727625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:31.727587 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:31:31.884525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:31.884487 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:32.102937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:32.100967 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:34.138070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:34.138033 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:31:34.186046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:34.186003 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:34.592246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:34.584696 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:34.805442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:34.805401 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:35.013054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:35.013020 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:36.186106 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:36.186075 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:31:39.049501 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.048824 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:39.138828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.138760 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" podUID="8c8dc548-e191-470b-8182-359426652ed6" containerName="payload-processing" containerID="cri-o://95cc8af29a2c16bc5212d63927bd7db62adec3e8aafe2207af6485936fc0869f" gracePeriod=25 Nov 02 15:31:39.394067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.394040 4469 generic.go:356] "Generic (PLEG): container finished" podID="8c8dc548-e191-470b-8182-359426652ed6" containerID="95cc8af29a2c16bc5212d63927bd7db62adec3e8aafe2207af6485936fc0869f" exitCode=0 Nov 02 15:31:39.394231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.394105 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" event={"ID":"8c8dc548-e191-470b-8182-359426652ed6","Type":"ContainerDied","Data":"95cc8af29a2c16bc5212d63927bd7db62adec3e8aafe2207af6485936fc0869f"} Nov 02 15:31:39.394231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.394172 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" event={"ID":"8c8dc548-e191-470b-8182-359426652ed6","Type":"ContainerDied","Data":"5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6"} Nov 02 15:31:39.394231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.394201 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5513a8bfdbdbce70126a710a1d243f1f72e863ebc9c3c0f6f22332d8df1512b6" Nov 02 15:31:39.411604 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.411578 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:31:39.588646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.588569 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\" (UniqueName: \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\") pod \"8c8dc548-e191-470b-8182-359426652ed6\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " Nov 02 15:31:39.588823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.588691 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\") pod \"8c8dc548-e191-470b-8182-359426652ed6\" (UID: \"8c8dc548-e191-470b-8182-359426652ed6\") " Nov 02 15:31:39.589099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.589077 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume" pod "8c8dc548-e191-470b-8182-359426652ed6" (UID: "8c8dc548-e191-470b-8182-359426652ed6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:39.590760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.590735 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm" pod "8c8dc548-e191-470b-8182-359426652ed6" (UID: "8c8dc548-e191-470b-8182-359426652ed6"). InnerVolumeSpecName "kube-api-access-xbqpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:39.690304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.690267 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqpm\" (UniqueName: \"kubernetes.io/projected/8c8dc548-e191-470b-8182-359426652ed6-kube-api-access-xbqpm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:39.690304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:39.690299 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8dc548-e191-470b-8182-359426652ed6-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:40.400250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:40.400218 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb" Nov 02 15:31:40.565876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:40.565841 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:31:40.615171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:40.615140 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-885c69-74b4b487c9-7nfnb"] Nov 02 15:31:41.239375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:41.239343 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8dc548-e191-470b-8182-359426652ed6" path="/var/lib/kubelet/pods/8c8dc548-e191-470b-8182-359426652ed6/volumes" Nov 02 15:31:43.813070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:43.813006 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:45.144597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:45.144552 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:48.052745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.052707 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:31:48.053613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.053578 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" podUID="4ec70061-42bd-4543-b001-3c4298b5ae78" containerName="maas-api" containerID="cri-o://f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac" gracePeriod=30 Nov 02 15:31:48.208022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.207988 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:48.377307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.377281 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:31:48.409636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.409609 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\") pod \"4ec70061-42bd-4543-b001-3c4298b5ae78\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " Nov 02 15:31:48.409833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.409713 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\" (UniqueName: \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\") pod \"4ec70061-42bd-4543-b001-3c4298b5ae78\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " Nov 02 15:31:48.409833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.409790 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") pod \"4ec70061-42bd-4543-b001-3c4298b5ae78\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " Nov 02 15:31:48.409974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.409859 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\") pod \"4ec70061-42bd-4543-b001-3c4298b5ae78\" (UID: \"4ec70061-42bd-4543-b001-3c4298b5ae78\") " Nov 02 15:31:48.410076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.410052 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca" pod "4ec70061-42bd-4543-b001-3c4298b5ae78" (UID: "4ec70061-42bd-4543-b001-3c4298b5ae78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:48.410535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.410514 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec70061-42bd-4543-b001-3c4298b5ae78-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:48.411898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.411872 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls" pod "4ec70061-42bd-4543-b001-3c4298b5ae78" (UID: "4ec70061-42bd-4543-b001-3c4298b5ae78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:31:48.412032 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.412004 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs" pod "4ec70061-42bd-4543-b001-3c4298b5ae78" (UID: "4ec70061-42bd-4543-b001-3c4298b5ae78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:31:48.412092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.412010 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n" pod "4ec70061-42bd-4543-b001-3c4298b5ae78" (UID: "4ec70061-42bd-4543-b001-3c4298b5ae78"). InnerVolumeSpecName "kube-api-access-w8f2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:48.507167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.507121 4469 generic.go:356] "Generic (PLEG): container finished" podID="4ec70061-42bd-4543-b001-3c4298b5ae78" containerID="f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac" exitCode=0 Nov 02 15:31:48.507365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.507351 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" event={"ID":"4ec70061-42bd-4543-b001-3c4298b5ae78","Type":"ContainerDied","Data":"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac"} Nov 02 15:31:48.507443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.507401 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" event={"ID":"4ec70061-42bd-4543-b001-3c4298b5ae78","Type":"ContainerDied","Data":"262d339bdbc145af59540667f59d88faf87c42ab380690b8e6354b1e5408933b"} Nov 02 15:31:48.507443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.507433 4469 scope.go:122] "RemoveContainer" containerID="f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac" Nov 02 15:31:48.507610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.507593 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb" Nov 02 15:31:48.514521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.514488 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8f2n\" (UniqueName: \"kubernetes.io/projected/4ec70061-42bd-4543-b001-3c4298b5ae78-kube-api-access-w8f2n\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:48.514521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.514522 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:48.514751 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.514538 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ec70061-42bd-4543-b001-3c4298b5ae78-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:31:48.530937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.530907 4469 scope.go:122] "RemoveContainer" containerID="f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac" Nov 02 15:31:48.531333 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:31:48.531312 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac\": container with ID starting with f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac not found: ID does not exist" containerID="f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac" Nov 02 15:31:48.531421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.531343 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac"} err="failed to get container status \"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac\": rpc error: code = NotFound desc = could not find container \"f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac\": container with ID starting with f8fb0fc0144c25940fa7a6a8ddd3a0e13389135cb15b6fa515709d6d1d96ceac not found: ID does not exist" Nov 02 15:31:48.682337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.682248 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:31:48.686504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:48.685687 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba49d0-6c5c659779-v8nlb"] Nov 02 15:31:49.240938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:49.240897 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec70061-42bd-4543-b001-3c4298b5ae78" path="/var/lib/kubelet/pods/4ec70061-42bd-4543-b001-3c4298b5ae78/volumes" Nov 02 15:31:49.731629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:49.731594 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:49.948995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:49.948959 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:50.184330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:50.184238 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:50.643017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:50.642977 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:55.017791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:55.017760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:55.225314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:55.225272 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:56.970392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:56.970345 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:57.180334 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.180299 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:57.230995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.230903 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:31:57.235211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.235152 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" containerID="cri-o://2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302" gracePeriod=30 Nov 02 15:31:57.242459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.242433 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:31:57.242603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.242587 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" containerID="cri-o://aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92" gracePeriod=30 Nov 02 15:31:57.388831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:31:57.388762 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:00.276105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:00.276069 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:00.276492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:00.276139 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:00.484894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:00.484852 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:00.846040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:00.846006 4469 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:00.846230 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:00.846066 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:02.543152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.543128 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:32:02.558940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.558920 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:32:02.601163 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.601134 4469 generic.go:356] "Generic (PLEG): container finished" podID="d0222cad-49d2-43f0-afad-e48e5d589046" containerID="2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302" exitCode=0 Nov 02 15:32:02.601351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.601183 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" Nov 02 15:32:02.601351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.601218 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerDied","Data":"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302"} Nov 02 15:32:02.601351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.601275 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk" event={"ID":"d0222cad-49d2-43f0-afad-e48e5d589046","Type":"ContainerDied","Data":"f77723b27a7d088576f476128ea0f9783c912d4ac3cf94a45196c77329bffabf"} Nov 02 15:32:02.601351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.601307 4469 scope.go:122] "RemoveContainer" containerID="2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302" Nov 02 15:32:02.602707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.602684 4469 generic.go:356] "Generic (PLEG): container finished" podID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerID="aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92" exitCode=0 Nov 02 15:32:02.602829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.602726 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" Nov 02 15:32:02.602829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.602763 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" event={"ID":"273869f1-0187-4321-af11-3ae9bbdfb6a7","Type":"ContainerDied","Data":"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92"} Nov 02 15:32:02.602931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.602827 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9" event={"ID":"273869f1-0187-4321-af11-3ae9bbdfb6a7","Type":"ContainerDied","Data":"93cb968cd9a11bdd89c8a47de3af04dfe1bf6ca61deb12c682cdb1258a0b4e6d"} Nov 02 15:32:02.610994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.610977 4469 scope.go:122] "RemoveContainer" containerID="71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d" Nov 02 15:32:02.619562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.619538 4469 scope.go:122] "RemoveContainer" containerID="2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302" Nov 02 15:32:02.619844 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:02.619827 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302\": container with ID starting with 2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302 not found: ID does not exist" containerID="2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302" Nov 02 15:32:02.619916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.619851 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302"} err="failed to get container status \"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302\": rpc error: code = NotFound desc = could not find container \"2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302\": container with ID starting with 2f26f6dadd32e485a4958bd72b94ca190c1b767b6fa4aca6cd4695e0bfbd7302 not found: ID does not exist" Nov 02 15:32:02.619916 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.619866 4469 scope.go:122] "RemoveContainer" containerID="71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d" Nov 02 15:32:02.620132 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:02.620114 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d\": container with ID starting with 71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d not found: ID does not exist" containerID="71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d" Nov 02 15:32:02.620198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.620142 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d"} err="failed to get container status \"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d\": rpc error: code = NotFound desc = could not find container \"71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d\": container with ID starting with 71715f56545f88b6f95a0e2e704cdc10d42453f837ca14c3d6746b971aa5921d not found: ID does not exist" Nov 02 15:32:02.620198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.620168 4469 scope.go:122] "RemoveContainer" containerID="aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92" Nov 02 15:32:02.628635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.628619 4469 scope.go:122] "RemoveContainer" containerID="aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92" Nov 02 15:32:02.628918 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:02.628900 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92\": container with ID starting with aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92 not found: ID does not exist" containerID="aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92" Nov 02 15:32:02.628973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.628923 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92"} err="failed to get container status \"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92\": rpc error: code = NotFound desc = could not find container \"aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92\": container with ID starting with aebc8411079fa1157fc8a0dda3bd1cec816c48bfb7270d68d7f8ac00b76c1e92 not found: ID does not exist" Nov 02 15:32:02.713019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.712987 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713096 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713134 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713183 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713204 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713220 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713271 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713304 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713339 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713367 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713394 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713402 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713408 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713439 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713499 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713534 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713600 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.713707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713675 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.714269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.713728 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:02.714647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.714622 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.714825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.714790 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\") pod \"d0222cad-49d2-43f0-afad-e48e5d589046\" (UID: \"d0222cad-49d2-43f0-afad-e48e5d589046\") " Nov 02 15:32:02.715030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.715012 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\") pod \"273869f1-0187-4321-af11-3ae9bbdfb6a7\" (UID: \"273869f1-0187-4321-af11-3ae9bbdfb6a7\") " Nov 02 15:32:02.715941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.715920 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717187 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717210 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717225 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0222cad-49d2-43f0-afad-e48e5d589046-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.716525 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.716571 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.716689 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.716886 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "kube-api-access-wfllc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717071 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.717297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717162 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.717819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.717315 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.719068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.719040 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:02.719165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.719088 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:02.720198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.720159 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.720198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.720164 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:02.720198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.720183 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7" pod "273869f1-0187-4321-af11-3ae9bbdfb6a7" (UID: "273869f1-0187-4321-af11-3ae9bbdfb6a7"). InnerVolumeSpecName "kube-api-access-n29n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:02.720677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.720652 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:02.721564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.721544 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy" pod "d0222cad-49d2-43f0-afad-e48e5d589046" (UID: "d0222cad-49d2-43f0-afad-e48e5d589046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818452 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818484 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818494 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0222cad-49d2-43f0-afad-e48e5d589046-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818503 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818513 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273869f1-0187-4321-af11-3ae9bbdfb6a7-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818526 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfllc\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-kube-api-access-wfllc\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818541 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29n7\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-kube-api-access-n29n7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818557 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818571 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0222cad-49d2-43f0-afad-e48e5d589046-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818579 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818590 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818599 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0222cad-49d2-43f0-afad-e48e5d589046-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818608 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273869f1-0187-4321-af11-3ae9bbdfb6a7-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.818920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:02.818616 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273869f1-0187-4321-af11-3ae9bbdfb6a7-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:03.028339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.028303 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:32:03.033486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.033459 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-sv6lk"] Nov 02 15:32:03.142567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.142474 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:32:03.146487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.146465 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-zsvr9"] Nov 02 15:32:03.248849 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.248774 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" path="/var/lib/kubelet/pods/273869f1-0187-4321-af11-3ae9bbdfb6a7/volumes" Nov 02 15:32:03.249520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:03.249501 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" path="/var/lib/kubelet/pods/d0222cad-49d2-43f0-afad-e48e5d589046/volumes" Nov 02 15:32:04.824238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:04.824199 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:06.186257 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:06.186226 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:32:06.476466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:06.476429 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:06.682790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:06.682758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:06.893320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:06.893235 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:07.098144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:07.098109 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:08.936480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:08.936442 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:12.265466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:12.265427 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:17.419724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:17.419689 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:17.629716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:17.629671 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:18.108618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:18.108583 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:32:21.839827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:21.839770 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:23.422677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:23.422641 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:23.646639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:23.646591 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:23.858427 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:23.858393 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:24.256644 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:24.256610 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:24.455006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:24.454971 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:24.673293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:24.673139 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:28.034565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:28.034529 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:29.383592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:29.383560 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:29.804463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:29.804419 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:30.095083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:30.094993 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:30.735368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:30.735340 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:31.144154 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:31.144111 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:31.410921 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:31.410840 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:31.639814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:31.639778 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:32.028693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:32.028652 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:34.949692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:34.949648 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:35.169493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:35.169456 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:36.186401 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:36.186364 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:32:36.566093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:36.566055 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:37.201856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:37.201821 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:37.416923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:37.416890 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:37.633956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:37.627084 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:37.873769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:37.873728 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:38.525464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:38.525432 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:38.749737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:38.749692 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:38.954135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:38.954100 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:39.167320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:39.167215 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:39.391824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:39.390791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:40.377517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:40.377482 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:40.657379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:40.657290 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:41.106161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.106127 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:32:41.106368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.106347 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" podUID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" containerName="maas-api" containerID="cri-o://59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5" gracePeriod=30 Nov 02 15:32:41.258795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.258737 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:41.400792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.400766 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:32:41.476414 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.476384 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\") pod \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " Nov 02 15:32:41.476556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.476437 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\") pod \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " Nov 02 15:32:41.476556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.476546 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\") pod \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " Nov 02 15:32:41.476628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.476606 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\" (UniqueName: \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\") pod \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\" (UID: \"fec05cde-9893-43e0-b2c7-1cb4e73849e0\") " Nov 02 15:32:41.476898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.476861 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca" pod "fec05cde-9893-43e0-b2c7-1cb4e73849e0" (UID: "fec05cde-9893-43e0-b2c7-1cb4e73849e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:41.477226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.477203 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec05cde-9893-43e0-b2c7-1cb4e73849e0-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:41.478608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.478586 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls" pod "fec05cde-9893-43e0-b2c7-1cb4e73849e0" (UID: "fec05cde-9893-43e0-b2c7-1cb4e73849e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:41.478685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.478587 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94" pod "fec05cde-9893-43e0-b2c7-1cb4e73849e0" (UID: "fec05cde-9893-43e0-b2c7-1cb4e73849e0"). InnerVolumeSpecName "kube-api-access-dhx94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:41.478685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.478628 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs" pod "fec05cde-9893-43e0-b2c7-1cb4e73849e0" (UID: "fec05cde-9893-43e0-b2c7-1cb4e73849e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:41.558540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.558328 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:41.578575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.578322 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:41.578575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.578358 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx94\" (UniqueName: \"kubernetes.io/projected/fec05cde-9893-43e0-b2c7-1cb4e73849e0-kube-api-access-dhx94\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:41.578575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.578379 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec05cde-9893-43e0-b2c7-1cb4e73849e0-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:32:41.781053 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.781021 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:41.930617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.930586 4469 generic.go:356] "Generic (PLEG): container finished" podID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" containerID="59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5" exitCode=0 Nov 02 15:32:41.930796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.930664 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" Nov 02 15:32:41.930796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.930680 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" event={"ID":"fec05cde-9893-43e0-b2c7-1cb4e73849e0","Type":"ContainerDied","Data":"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5"} Nov 02 15:32:41.930796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.930751 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m" event={"ID":"fec05cde-9893-43e0-b2c7-1cb4e73849e0","Type":"ContainerDied","Data":"8d2fb15c95b0d6f7fbabd3380512195f25e2011b2f3cd331be7e7cee5d347e54"} Nov 02 15:32:41.930796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.930783 4469 scope.go:122] "RemoveContainer" containerID="59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5" Nov 02 15:32:41.943103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.942698 4469 scope.go:122] "RemoveContainer" containerID="59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5" Nov 02 15:32:41.943103 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:32:41.943058 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5\": container with ID starting with 59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5 not found: ID does not exist" containerID="59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5" Nov 02 15:32:41.943313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.943092 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5"} err="failed to get container status \"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5\": rpc error: code = NotFound desc = could not find container \"59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5\": container with ID starting with 59344789c0cad35cc3e64a0281a9de4769732d72eff2bc0c2988a38b7d36bea5 not found: ID does not exist" Nov 02 15:32:41.990580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:41.990550 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:42.103842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:42.103711 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:32:42.113922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:42.113887 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b87e90-7468dc88b4-xtt5m"] Nov 02 15:32:43.023994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:43.023958 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:43.240047 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:43.240010 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" path="/var/lib/kubelet/pods/fec05cde-9893-43e0-b2c7-1cb4e73849e0/volumes" Nov 02 15:32:43.245864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:43.245828 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:43.447273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:43.447177 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:43.893570 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:43.893539 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:44.123872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:44.123737 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:45.472885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:45.472843 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:47.779538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:47.779498 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:49.366488 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:49.366455 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:49.583026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:49.582992 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:49.791787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:49.791749 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:50.002164 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:50.002122 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:50.394245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:50.394212 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:50.840613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:50.840563 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:51.065966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:51.065934 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:51.272398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:51.272317 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:52.319293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:52.319262 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:32:54.985547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:54.985508 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:55.464458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:55.464431 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:57.694335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:57.694280 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:57.901444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:57.901414 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:57.959399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:57.959368 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:32:57.959671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:57.959618 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" containerID="cri-o://7eed4cbdc2b29deaf3e5666e45e446c00276fa667c3d106024e5350636ecc155" gracePeriod=30 Nov 02 15:32:57.976001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:57.975976 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:32:58.003385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.003359 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:32:58.003568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.003547 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" containerID="cri-o://da04aa03d2f8266ca54c784ce9e8b3ddf742b232e77dc8c61b0dfa48cf93595b" gracePeriod=30 Nov 02 15:32:58.104024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.103977 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" containerID="cri-o://8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce" gracePeriod=30 Nov 02 15:32:58.111438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.111404 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:58.679016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.678960 4469 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:58.679180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:58.679051 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:59.723404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:32:59.723368 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:00.270043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:00.270006 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:03.142895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.142860 4469 generic.go:356] "Generic (PLEG): container finished" podID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerID="da04aa03d2f8266ca54c784ce9e8b3ddf742b232e77dc8c61b0dfa48cf93595b" exitCode=0 Nov 02 15:33:03.143771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.142929 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerDied","Data":"da04aa03d2f8266ca54c784ce9e8b3ddf742b232e77dc8c61b0dfa48cf93595b"} Nov 02 15:33:03.143771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.143644 4469 scope.go:122] "RemoveContainer" containerID="ca15cdbf6e3c5ab0fddf0ac7da86c7709e89fbb7d5b3799cdd8e9da2e75596cf" Nov 02 15:33:03.146516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.146482 4469 generic.go:356] "Generic (PLEG): container finished" podID="fa05a93f-620a-4649-b108-f2dce51efb36" containerID="7eed4cbdc2b29deaf3e5666e45e446c00276fa667c3d106024e5350636ecc155" exitCode=0 Nov 02 15:33:03.146584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.146553 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" event={"ID":"fa05a93f-620a-4649-b108-f2dce51efb36","Type":"ContainerDied","Data":"7eed4cbdc2b29deaf3e5666e45e446c00276fa667c3d106024e5350636ecc155"} Nov 02 15:33:03.168319 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:33:03.168290 4469 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c49285_8334_4e73_8a3a_9af5076e55cd.slice/crio-conmon-8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:33:03.318462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.318432 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:33:03.328673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.328646 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:33:03.365281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365252 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365289 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365422 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365346 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365437 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365530 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365539 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.365664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365588 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365660 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365697 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.365845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.365729 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\") pod \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\" (UID: \"9ba1d179-ef29-433d-9a5f-5916d7afd14a\") " Nov 02 15:33:03.366152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.366129 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.367637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.366644 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.367637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.366672 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.367637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.367006 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:03.367637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.367038 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.367637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.367288 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.368104 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.368066 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.368236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.368211 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.369555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.369528 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:33:03.369902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.369877 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg" pod "9ba1d179-ef29-433d-9a5f-5916d7afd14a" (UID: "9ba1d179-ef29-433d-9a5f-5916d7afd14a"). InnerVolumeSpecName "kube-api-access-pjrcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.377737 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.377706 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:33:03.467420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467392 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467428 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467450 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467471 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467497 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467524 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467564 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467588 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467637 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467668 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467694 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467746 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467830 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467888 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.467912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467904 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467935 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467992 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\") pod \"fa05a93f-620a-4649-b108-f2dce51efb36\" (UID: \"fa05a93f-620a-4649-b108-f2dce51efb36\") " Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.467990 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468056 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468098 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468115 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\") pod \"f8c49285-8334-4e73-8a3a-9af5076e55cd\" (UID: \"f8c49285-8334-4e73-8a3a-9af5076e55cd\") " Nov 02 15:33:03.468359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468284 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.468748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468455 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.468748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468672 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468921 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468946 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468962 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468980 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.468995 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469009 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrcg\" (UniqueName: \"kubernetes.io/projected/9ba1d179-ef29-433d-9a5f-5916d7afd14a-kube-api-access-pjrcg\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469024 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469037 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469050 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05a93f-620a-4649-b108-f2dce51efb36-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469066 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba1d179-ef29-433d-9a5f-5916d7afd14a-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469080 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469094 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.469226 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469107 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.470009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469300 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.470009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469410 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.470009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469508 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.470009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.469598 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:03.470321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.470282 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.470699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.470661 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.470861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.470768 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:33:03.470937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.470875 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:03.471103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.471079 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4" pod "f8c49285-8334-4e73-8a3a-9af5076e55cd" (UID: "f8c49285-8334-4e73-8a3a-9af5076e55cd"). InnerVolumeSpecName "kube-api-access-7zxm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.471169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.471154 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "kube-api-access-mczgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.471287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.471273 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:33:03.471552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.471539 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token" pod "fa05a93f-620a-4649-b108-f2dce51efb36" (UID: "fa05a93f-620a-4649-b108-f2dce51efb36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:03.570819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570724 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.570819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570760 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.570819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570778 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.570819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570791 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570821 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570836 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczgs\" (UniqueName: \"kubernetes.io/projected/fa05a93f-620a-4649-b108-f2dce51efb36-kube-api-access-mczgs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570850 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05a93f-620a-4649-b108-f2dce51efb36-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570864 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570879 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c49285-8334-4e73-8a3a-9af5076e55cd-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570909 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c49285-8334-4e73-8a3a-9af5076e55cd-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570923 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxm4\" (UniqueName: \"kubernetes.io/projected/f8c49285-8334-4e73-8a3a-9af5076e55cd-kube-api-access-7zxm4\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.571097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.570938 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05a93f-620a-4649-b108-f2dce51efb36-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:03.868875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:03.860905 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:04.069936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.069900 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:04.153619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.153543 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" Nov 02 15:33:04.153619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.153579 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq" event={"ID":"9ba1d179-ef29-433d-9a5f-5916d7afd14a","Type":"ContainerDied","Data":"52b46b619b65c08ee5068d024e1b00b1e064abe715a13eb9327b4ddc83effe37"} Nov 02 15:33:04.154184 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.153643 4469 scope.go:122] "RemoveContainer" containerID="da04aa03d2f8266ca54c784ce9e8b3ddf742b232e77dc8c61b0dfa48cf93595b" Nov 02 15:33:04.155201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.155178 4469 generic.go:356] "Generic (PLEG): container finished" podID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerID="8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce" exitCode=0 Nov 02 15:33:04.155273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.155227 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" Nov 02 15:33:04.155315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.155269 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" event={"ID":"f8c49285-8334-4e73-8a3a-9af5076e55cd","Type":"ContainerDied","Data":"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce"} Nov 02 15:33:04.155367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.155321 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj" event={"ID":"f8c49285-8334-4e73-8a3a-9af5076e55cd","Type":"ContainerDied","Data":"bf1942fda7acdfee4979e27106ad399c2066e3bf909e6025313b744269b79e1b"} Nov 02 15:33:04.156571 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.156549 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" event={"ID":"fa05a93f-620a-4649-b108-f2dce51efb36","Type":"ContainerDied","Data":"ab3b6983e978879a2aa6977807c1879fd26ca0c5f2b0cd02631f725ad132646d"} Nov 02 15:33:04.156656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.156636 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4" Nov 02 15:33:04.174165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.174129 4469 scope.go:122] "RemoveContainer" containerID="8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce" Nov 02 15:33:04.184386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.184365 4469 scope.go:122] "RemoveContainer" containerID="8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce" Nov 02 15:33:04.184753 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:33:04.184717 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce\": container with ID starting with 8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce not found: ID does not exist" containerID="8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce" Nov 02 15:33:04.184867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.184780 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce"} err="failed to get container status \"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce\": rpc error: code = NotFound desc = could not find container \"8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce\": container with ID starting with 8e1e4ea82946cfeaac544bb0edc331bc674b927842b78de1dca4444a6783a5ce not found: ID does not exist" Nov 02 15:33:04.184941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.184859 4469 scope.go:122] "RemoveContainer" containerID="7eed4cbdc2b29deaf3e5666e45e446c00276fa667c3d106024e5350636ecc155" Nov 02 15:33:04.320177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.320140 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:33:04.369026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.368995 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-d9msj"] Nov 02 15:33:04.431754 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.431677 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:33:04.435922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.435898 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-fzgw4"] Nov 02 15:33:04.544915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.544883 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:33:04.552186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:04.552158 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-ntwpq"] Nov 02 15:33:05.246686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:05.246652 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" path="/var/lib/kubelet/pods/9ba1d179-ef29-433d-9a5f-5916d7afd14a/volumes" Nov 02 15:33:05.247990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:05.247970 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" path="/var/lib/kubelet/pods/f8c49285-8334-4e73-8a3a-9af5076e55cd/volumes" Nov 02 15:33:05.248602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:05.248588 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" path="/var/lib/kubelet/pods/fa05a93f-620a-4649-b108-f2dce51efb36/volumes" Nov 02 15:33:05.404822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:05.403719 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:06.188040 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:33:06.188005 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:33:06.189791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:06.189765 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:09.288356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:09.288322 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:10.376989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:10.376954 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:11.310962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:11.310931 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:11.623755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:11.623666 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:12.034859 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:12.034791 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:13.460317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:13.460282 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:13.670146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:13.670114 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:15.184290 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:15.184252 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:33:15.184722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:15.184537 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" containerID="cri-o://0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae" gracePeriod=30 Nov 02 15:33:15.650476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:15.650391 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:17.370652 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:17.370610 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:17.583394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:17.583360 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:18.310526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:18.310461 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:18.319285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:18.319258 4469 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:33:18.319404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:18.319311 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:33:20.455049 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.454968 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:33:20.497602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497570 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497642 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497683 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497773 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497828 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:20.497959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497854 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497912 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.497959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497948 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.498183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.497987 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.498183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498016 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\") pod \"9ce93c0b-c744-4e39-821d-2026121c2233\" (UID: \"9ce93c0b-c744-4e39-821d-2026121c2233\") " Nov 02 15:33:20.498183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498084 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:20.498356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498182 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:20.498356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498303 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498544 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498700 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498722 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498754 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498771 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ce93c0b-c744-4e39-821d-2026121c2233-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.498824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.498786 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.499988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.499964 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:33:20.500109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.500086 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "kube-api-access-lvqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:20.500185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.500117 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:20.500373 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.500357 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy" pod "9ce93c0b-c744-4e39-821d-2026121c2233" (UID: "9ce93c0b-c744-4e39-821d-2026121c2233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:20.600123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.600091 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.600123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.600122 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ce93c0b-c744-4e39-821d-2026121c2233-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.600484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.600137 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ce93c0b-c744-4e39-821d-2026121c2233-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.600484 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.600152 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqgc\" (UniqueName: \"kubernetes.io/projected/9ce93c0b-c744-4e39-821d-2026121c2233-kube-api-access-lvqgc\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:33:20.601280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.601251 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:20.805228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:20.805185 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:21.016933 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.016895 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:21.128291 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.128194 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:33:21.129370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129345 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.129531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129516 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.129624 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129613 4469 cpu_manager.go:414] "containerMap: removing container" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" Nov 02 15:33:21.129710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129698 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" Nov 02 15:33:21.129816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129787 4469 cpu_manager.go:414] "containerMap: removing container" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.129949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.129935 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.130072 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130059 4469 cpu_manager.go:414] "containerMap: removing container" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" Nov 02 15:33:21.130165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130153 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" Nov 02 15:33:21.130323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130309 4469 cpu_manager.go:414] "containerMap: removing container" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" Nov 02 15:33:21.130455 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130443 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" Nov 02 15:33:21.130531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130521 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.130603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130593 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.130678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130669 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.130742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130734 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.130854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130844 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.130995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.130983 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.131077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131067 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" containerName="payload-processing" Nov 02 15:33:21.131196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131184 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" containerName="payload-processing" Nov 02 15:33:21.131268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131259 4469 cpu_manager.go:414] "containerMap: removing container" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.131343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131332 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.131445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131428 4469 cpu_manager.go:414] "containerMap: removing container" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.131553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131544 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.131631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131623 4469 cpu_manager.go:414] "containerMap: removing container" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" Nov 02 15:33:21.131706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131696 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" Nov 02 15:33:21.131782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131773 4469 cpu_manager.go:414] "containerMap: removing container" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.131877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131866 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.131956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.131947 4469 cpu_manager.go:414] "containerMap: removing container" podUID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" containerName="maas-api" Nov 02 15:33:21.132029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132017 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" containerName="maas-api" Nov 02 15:33:21.132119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132109 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:33:21.132189 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132180 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:33:21.132264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132255 4469 cpu_manager.go:414] "containerMap: removing container" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.132337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132326 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.132406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132398 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.132477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132467 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.132559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132550 4469 cpu_manager.go:414] "containerMap: removing container" podUID="4ec70061-42bd-4543-b001-3c4298b5ae78" containerName="maas-api" Nov 02 15:33:21.132633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132624 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec70061-42bd-4543-b001-3c4298b5ae78" containerName="maas-api" Nov 02 15:33:21.132715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132705 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" Nov 02 15:33:21.132792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132778 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" Nov 02 15:33:21.132939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.132918 4469 cpu_manager.go:414] "containerMap: removing container" podUID="5692e221-7431-4fe8-9d4b-9a6afe036821" containerName="payload-pre-processing" Nov 02 15:33:21.133022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133010 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5692e221-7431-4fe8-9d4b-9a6afe036821" containerName="payload-pre-processing" Nov 02 15:33:21.133109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133099 4469 cpu_manager.go:414] "containerMap: removing container" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" Nov 02 15:33:21.133225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133204 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" Nov 02 15:33:21.133381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133367 4469 cpu_manager.go:414] "containerMap: removing container" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.133504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133482 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.133608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133586 4469 cpu_manager.go:414] "containerMap: removing container" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:33:21.133608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133602 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133615 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133622 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133636 4469 cpu_manager.go:414] "containerMap: removing container" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133643 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133656 4469 cpu_manager.go:414] "containerMap: removing container" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133663 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133674 4469 cpu_manager.go:414] "containerMap: removing container" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133681 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133693 4469 cpu_manager.go:414] "containerMap: removing container" podUID="8c8dc548-e191-470b-8182-359426652ed6" containerName="payload-processing" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133701 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8dc548-e191-470b-8182-359426652ed6" containerName="payload-processing" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133711 4469 cpu_manager.go:414] "containerMap: removing container" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133718 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133727 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.133741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133733 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133748 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133755 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133766 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133775 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133786 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133792 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133820 4469 cpu_manager.go:414] "containerMap: removing container" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133826 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133842 4469 cpu_manager.go:414] "containerMap: removing container" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133849 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133860 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133867 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133883 4469 cpu_manager.go:414] "containerMap: removing container" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133889 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133910 4469 cpu_manager.go:414] "containerMap: removing container" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133918 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133930 4469 cpu_manager.go:414] "containerMap: removing container" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133937 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133951 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133958 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133971 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133979 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133990 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.133996 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134008 4469 cpu_manager.go:414] "containerMap: removing container" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134017 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134028 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134035 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134045 4469 cpu_manager.go:414] "containerMap: removing container" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134051 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134062 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134068 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134090 4469 cpu_manager.go:414] "containerMap: removing container" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134098 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134113 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134120 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134130 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134137 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134287 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134300 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf67fec-d305-44f3-ad86-73424427b88e" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134312 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="273869f1-0187-4321-af11-3ae9bbdfb6a7" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134323 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134334 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134343 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134352 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134360 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134370 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134379 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134388 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134399 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171aa1f-3ac5-4dcb-ad8b-8e2dc19658c9" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134410 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b158bdd3-ecc6-460f-96b7-0753f3aa8c1d" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134419 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f58fc1b-bcc0-4a74-a2c3-95d224259fcd" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134428 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d759e268-21dd-4530-9a59-780148841f93" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134438 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c1ebc6-79ba-4e00-886c-e044eec5ecb4" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134447 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134457 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c49285-8334-4e73-8a3a-9af5076e55cd" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134467 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="7697f54a-9b62-46cf-aa9d-aa753630d01b" containerName="main" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134477 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134485 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134493 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e06e67-6bf0-4d82-8fcd-69a8170cf291" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134504 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134514 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134520 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa05a93f-620a-4649-b108-f2dce51efb36" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134528 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134537 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec05cde-9893-43e0-b2c7-1cb4e73849e0" containerName="maas-api" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134545 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ba386c-d24f-480d-ad3b-5f66d742c4c4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134553 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134562 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec70061-42bd-4543-b001-3c4298b5ae78" containerName="maas-api" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134570 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dbd84f-4471-454d-89d3-48f756e99e51" containerName="main" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134580 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="5692e221-7431-4fe8-9d4b-9a6afe036821" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134590 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="44660d8c-1252-4503-8303-afffee9fb66d" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134601 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="11abbca7-0516-43c8-9547-954453117952" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134610 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134619 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="7030d72e-a96f-420d-9d3f-a373b196c485" containerName="istio-proxy" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134626 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8dc548-e191-470b-8182-359426652ed6" containerName="payload-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134634 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134641 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134648 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c464c645-62d1-4a0c-a83e-83a51d1bf408" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134659 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6397750-8c5c-4ccf-bbf7-e88fbd23eb85" containerName="payload-pre-processing" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134667 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:33:21.134629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134676 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa67505-1d76-438f-b61d-81b0bd53f704" containerName="main" Nov 02 15:33:21.138346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.134687 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6b9172-a97d-49a0-a6ff-f4d126ad54d3" containerName="cleanup" Nov 02 15:33:21.139635 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.139614 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.155533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.155509 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:33:21.191970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.191933 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4e4683\"" Nov 02 15:33:21.192149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.192115 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4e4683-dockercfg-8whlx\"" Nov 02 15:33:21.220453 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.220428 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:21.223043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.223015 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.223162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.223053 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbpx\" (UniqueName: \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.266761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.266736 4469 generic.go:356] "Generic (PLEG): container finished" podID="9ce93c0b-c744-4e39-821d-2026121c2233" containerID="0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae" exitCode=0 Nov 02 15:33:21.266918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.266780 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" Nov 02 15:33:21.266918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.266841 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" event={"ID":"9ce93c0b-c744-4e39-821d-2026121c2233","Type":"ContainerDied","Data":"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae"} Nov 02 15:33:21.266918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.266895 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8" event={"ID":"9ce93c0b-c744-4e39-821d-2026121c2233","Type":"ContainerDied","Data":"402e4276a9c3ee524d84d6f19a24262c0ac0370f3038695bc5916bd8faa6ea53"} Nov 02 15:33:21.266918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.266919 4469 scope.go:122] "RemoveContainer" containerID="0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae" Nov 02 15:33:21.276383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.276367 4469 scope.go:122] "RemoveContainer" containerID="0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae" Nov 02 15:33:21.276613 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:33:21.276596 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae\": container with ID starting with 0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae not found: ID does not exist" containerID="0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae" Nov 02 15:33:21.276669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.276620 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae"} err="failed to get container status \"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae\": rpc error: code = NotFound desc = could not find container \"0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae\": container with ID starting with 0fd46ab2424e88c146c66270cd9d9f10970859d6777f2dc77c0ec2ee2b7483ae not found: ID does not exist" Nov 02 15:33:21.323899 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.323867 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.324011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.323907 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbpx\" (UniqueName: \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.324510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.324487 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.384597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.384529 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbpx\" (UniqueName: \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\") pod \"payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.432594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.432557 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:33:21.438497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.438470 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-lrjs8"] Nov 02 15:33:21.454154 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.454129 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:21.831298 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:21.831273 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:33:21.833563 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:33:21.833537 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8626d5_72ed_47af_b0f8_2b0863811b8e.slice/crio-2127ed4e3d6fe7400f03c14930b7c5d599733cc3b9c3126e0f0800ed2b530e37 WatchSource:0}: Error finding container 2127ed4e3d6fe7400f03c14930b7c5d599733cc3b9c3126e0f0800ed2b530e37: Status 404 returned error can't find the container with id 2127ed4e3d6fe7400f03c14930b7c5d599733cc3b9c3126e0f0800ed2b530e37 Nov 02 15:33:22.274727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:22.274690 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" event={"ID":"cf8626d5-72ed-47af-b0f8-2b0863811b8e","Type":"ContainerStarted","Data":"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183"} Nov 02 15:33:22.274968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:22.274750 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" event={"ID":"cf8626d5-72ed-47af-b0f8-2b0863811b8e","Type":"ContainerStarted","Data":"2127ed4e3d6fe7400f03c14930b7c5d599733cc3b9c3126e0f0800ed2b530e37"} Nov 02 15:33:22.274968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:22.274784 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:22.392657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:22.392599 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" podStartSLOduration=1.392586329 podStartE2EDuration="1.392586329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:22.390840177 +0000 UTC m=+2205.274442694" watchObservedRunningTime="2026-11-02 15:33:22.392586329 +0000 UTC m=+2205.276188882" Nov 02 15:33:23.239319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:23.239289 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce93c0b-c744-4e39-821d-2026121c2233" path="/var/lib/kubelet/pods/9ce93c0b-c744-4e39-821d-2026121c2233/volumes" Nov 02 15:33:25.827557 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:25.827243 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:30.566823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:30.565789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:32.309654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:32.309617 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:33:33.285054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:33.285022 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:33:35.788825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:35.786733 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:36.188003 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:33:36.187911 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:33:38.378512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:38.378479 4469 scope.go:122] "RemoveContainer" containerID="95cc8af29a2c16bc5212d63927bd7db62adec3e8aafe2207af6485936fc0869f" Nov 02 15:33:40.674753 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:40.674716 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:44.860239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:44.860197 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:45.839658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:45.839616 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:46.422373 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:46.422331 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:46.711146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:46.711112 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:46.980598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:46.980503 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:47.341833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:47.341728 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:51.864585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:51.864549 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:53.722129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:53.722093 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.009348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:54.009253 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.253212 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:54.253175 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.553547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:54.553507 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:55.853356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:55.853321 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:56.864562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:56.859301 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:59.548245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:59.548209 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:33:59.597726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:59.597686 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:33:59.773007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:33:59.772972 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:02.191137 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:02.191096 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:06.125418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:06.125381 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:34:06.188245 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:34:06.188208 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:34:06.523795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:06.523762 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:34:06.634238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:06.634126 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:34:06.961872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:06.961764 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:07.172028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:07.171980 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:07.393407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:07.393308 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:07.602599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:07.602560 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:07.809903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:07.809868 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:09.679521 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:09.679484 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:11.524871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:11.524778 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" containerID="cri-o://4e368d5db8e9a7d49f5a12dd565b74cb6b2899359d2b39c5e3d4919357456394" gracePeriod=25 Nov 02 15:34:11.634756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:11.634710 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" containerID="cri-o://ec01e8a7d3bbc80e82dff2e2f3b051570beedb6504329c03910ea538a7f4c5a7" gracePeriod=25 Nov 02 15:34:12.556589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:12.556550 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:13.201463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:13.201425 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:34:14.570059 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.570014 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" containerID="cri-o://b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806" gracePeriod=45 Nov 02 15:34:14.619969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.619913 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" containerID="cri-o://b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4" gracePeriod=45 Nov 02 15:34:14.910940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.910907 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:34:14.913303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.913282 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:34:14.989045 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989011 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.989206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989073 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.989206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989127 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.989206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989195 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.989391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989231 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.989391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989282 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\" (UniqueName: \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.989391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989299 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.989391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989355 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.989391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989364 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989392 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989402 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989458 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\" (UniqueName: \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989499 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989525 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989550 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.989658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989556 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\") pod \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\" (UID: \"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b\") " Nov 02 15:34:14.990012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989681 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\") pod \"e406796a-50ff-4cdb-a7be-c90bf453374a\" (UID: \"e406796a-50ff-4cdb-a7be-c90bf453374a\") " Nov 02 15:34:14.990012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.989984 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.990308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990284 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.990400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990311 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.990400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990327 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.990400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990341 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.990400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990354 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.990400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.990368 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:14.993144 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993109 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:14.993238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993203 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.993405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993376 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:14.993513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993445 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv" pod "e406796a-50ff-4cdb-a7be-c90bf453374a" (UID: "e406796a-50ff-4cdb-a7be-c90bf453374a"). InnerVolumeSpecName "kube-api-access-zdxgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:14.993582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993511 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:14.993582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:14.993520 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf" pod "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" (UID: "ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b"). InnerVolumeSpecName "kube-api-access-hfwwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:15.091887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091855 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxgv\" (UniqueName: \"kubernetes.io/projected/e406796a-50ff-4cdb-a7be-c90bf453374a-kube-api-access-zdxgv\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.091887 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091886 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.092098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091900 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e406796a-50ff-4cdb-a7be-c90bf453374a-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.092098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091912 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwwf\" (UniqueName: \"kubernetes.io/projected/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-kube-api-access-hfwwf\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.092098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091925 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.092098 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.091937 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e406796a-50ff-4cdb-a7be-c90bf453374a-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:15.656125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.656039 4469 generic.go:356] "Generic (PLEG): container finished" podID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerID="b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806" exitCode=0 Nov 02 15:34:15.656125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.656090 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" Nov 02 15:34:15.656633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.656129 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" event={"ID":"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b","Type":"ContainerDied","Data":"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806"} Nov 02 15:34:15.656633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.656178 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg" event={"ID":"ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b","Type":"ContainerDied","Data":"a1acccab19cc03ceaaef94a3b86340b2e1f23296bc9ec680a86c445bf2750257"} Nov 02 15:34:15.656633 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.656195 4469 scope.go:122] "RemoveContainer" containerID="b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806" Nov 02 15:34:15.657591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.657568 4469 generic.go:356] "Generic (PLEG): container finished" podID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerID="b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4" exitCode=0 Nov 02 15:34:15.657663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.657599 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" Nov 02 15:34:15.657663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.657642 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" event={"ID":"e406796a-50ff-4cdb-a7be-c90bf453374a","Type":"ContainerDied","Data":"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4"} Nov 02 15:34:15.657752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.657678 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" event={"ID":"e406796a-50ff-4cdb-a7be-c90bf453374a","Type":"ContainerDied","Data":"86d09d6858256af757125f8684e42be9583b2367b3ee23b65af9bea54c025c00"} Nov 02 15:34:15.666222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.666200 4469 scope.go:122] "RemoveContainer" containerID="b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806" Nov 02 15:34:15.666512 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:34:15.666486 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806\": container with ID starting with b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806 not found: ID does not exist" containerID="b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806" Nov 02 15:34:15.666597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.666521 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806"} err="failed to get container status \"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806\": rpc error: code = NotFound desc = could not find container \"b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806\": container with ID starting with b8a4f073bd67ec4f2c025865d3dc2ee2734a65a3c9a7542b2e05aef601ace806 not found: ID does not exist" Nov 02 15:34:15.666597 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.666539 4469 scope.go:122] "RemoveContainer" containerID="b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4" Nov 02 15:34:15.675275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.675259 4469 scope.go:122] "RemoveContainer" containerID="b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4" Nov 02 15:34:15.675543 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:34:15.675523 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4\": container with ID starting with b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4 not found: ID does not exist" containerID="b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4" Nov 02 15:34:15.675601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.675550 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4"} err="failed to get container status \"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4\": rpc error: code = NotFound desc = could not find container \"b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4\": container with ID starting with b7b1db79ebf3b6ea5d8a34e8677b53a2763cb0b19ea2ec5b1747e9c2a3981dc4 not found: ID does not exist" Nov 02 15:34:15.832764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.832729 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:34:15.881598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.881560 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-2-w3-42555d-kserve-6bc9599b6d-5lsbg"] Nov 02 15:34:15.887946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.887912 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:15.991158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.991122 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:34:15.993567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:15.993545 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/distinct-w3-42555d-kserve-75f67c9f9-56wxg"] Nov 02 15:34:17.238892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:17.238861 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" path="/var/lib/kubelet/pods/ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b/volumes" Nov 02 15:34:17.239288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:17.239273 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" path="/var/lib/kubelet/pods/e406796a-50ff-4cdb-a7be-c90bf453374a/volumes" Nov 02 15:34:17.367285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:17.367247 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:19.614689 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.614659 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:34:19.615216 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615202 4469 cpu_manager.go:414] "containerMap: removing container" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" Nov 02 15:34:19.615272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615217 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" Nov 02 15:34:19.615272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615229 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" Nov 02 15:34:19.615272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615235 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" Nov 02 15:34:19.615272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615272 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615277 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615352 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e406796a-50ff-4cdb-a7be-c90bf453374a" containerName="main" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615359 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a298d4d4-4178-4ee1-995b-707345440cf4" containerName="cleanup" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615369 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b036f705-7118-4c89-82ab-92f321ebc7e9" containerName="cleanup" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615377 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba1d179-ef29-433d-9a5f-5916d7afd14a" containerName="istio-proxy" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615383 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0222cad-49d2-43f0-afad-e48e5d589046" containerName="istio-proxy" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615389 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7defa54-df39-4771-b93b-209602778580" containerName="istio-proxy" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615396 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b00d3-3f96-4521-936c-e47dde2fe6e8" containerName="cleanup" Nov 02 15:34:19.615410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.615406 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac70e2e1-bf7b-4da3-ba9e-a76b86724a5b" containerName="main" Nov 02 15:34:19.620225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.620206 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.742260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.742227 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:34:19.751602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751567 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.751777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751640 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.751777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751676 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.751777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751762 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.752026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751817 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.752026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751846 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zm76\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.752026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751886 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.752026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751912 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.752026 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.751983 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853489 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853597 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853629 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853646 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853716 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853751 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853840 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zm76\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853902 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.853989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.853932 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.854206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.854016 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.854206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.854054 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.854374 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.854348 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.854425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.854390 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.854467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.854422 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.856248 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.856228 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.856482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.856461 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.918721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.918659 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.918994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.918972 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zm76\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:19.933583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:19.933555 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:20.370677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.370651 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:34:20.373629 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:34:20.373601 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8babd2ea_5857_429f_8748_d75181d42d86.slice/crio-ad81836cd52432b8c5d266c6c3f9c3977d87c559dcbf7f5011c7fe71f6b0c822 WatchSource:0}: Error finding container ad81836cd52432b8c5d266c6c3f9c3977d87c559dcbf7f5011c7fe71f6b0c822: Status 404 returned error can't find the container with id ad81836cd52432b8c5d266c6c3f9c3977d87c559dcbf7f5011c7fe71f6b0c822 Nov 02 15:34:20.376936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.376902 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:20.377020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.376969 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:20.377020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.377000 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:20.693839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.693722 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" event={"ID":"8babd2ea-5857-429f-8748-d75181d42d86","Type":"ContainerStarted","Data":"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e"} Nov 02 15:34:20.693839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.693783 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" event={"ID":"8babd2ea-5857-429f-8748-d75181d42d86","Type":"ContainerStarted","Data":"ad81836cd52432b8c5d266c6c3f9c3977d87c559dcbf7f5011c7fe71f6b0c822"} Nov 02 15:34:20.824794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.824731 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podStartSLOduration=1.824712335 podStartE2EDuration="1.824712335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:20.821361028 +0000 UTC m=+2263.704963548" watchObservedRunningTime="2026-11-02 15:34:20.824712335 +0000 UTC m=+2263.708314857" Nov 02 15:34:20.933696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:20.933669 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:21.187525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:21.187488 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:34:21.949979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:21.949943 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:34:21.950568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:21.950537 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:34:22.418826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:22.418765 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:22.942209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:22.942170 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:34:22.943280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:22.943246 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:34:23.201585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:23.201497 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:34:23.558739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:23.558686 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:23.941861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:23.940908 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:34:23.941861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:23.940997 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:34:24.943237 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:24.943198 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:34:24.943850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:24.943819 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:34:25.945665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:25.945240 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:34:25.945665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:25.945297 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:34:27.392436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:27.392403 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:27.392917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:27.392705 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:27.393870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:27.393852 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:34:27.694770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:27.694632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:30.431608 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.431568 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc"] Nov 02 15:34:30.436056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.436030 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.494120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.494089 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc"] Nov 02 15:34:30.515266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.515239 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.515402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.515279 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqnd\" (UniqueName: \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.515444 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.515397 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.540228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.540199 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:34:30.540433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.540253 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:34:30.616358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.616326 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.616511 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.616372 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqnd\" (UniqueName: \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.616511 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.616462 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.617105 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.617083 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.618606 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.618584 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.677759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.677727 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqnd\" (UniqueName: \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\") pod \"maas-api-revoke-sub-1f05f9fc-rrkmc\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:30.751238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:30.751173 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:31.128835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.128794 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc"] Nov 02 15:34:31.131475 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:34:31.131448 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94193325_ecad_4c9d_949b_4c1e83cda24f.slice/crio-096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96 WatchSource:0}: Error finding container 096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96: Status 404 returned error can't find the container with id 096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96 Nov 02 15:34:31.188139 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.188108 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:34:31.766381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.766228 4469 generic.go:356] "Generic (PLEG): container finished" podID="ab4c08af-9592-414b-945e-6497bc057e57" containerID="4e368d5db8e9a7d49f5a12dd565b74cb6b2899359d2b39c5e3d4919357456394" exitCode=0 Nov 02 15:34:31.766381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.766263 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" event={"ID":"ab4c08af-9592-414b-945e-6497bc057e57","Type":"ContainerDied","Data":"4e368d5db8e9a7d49f5a12dd565b74cb6b2899359d2b39c5e3d4919357456394"} Nov 02 15:34:31.769500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.768507 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" event={"ID":"94193325-ecad-4c9d-949b-4c1e83cda24f","Type":"ContainerStarted","Data":"6cbe997421e9ec7c5b6a3bb3eb072ba7d06e884d259d542d7e8bbceb949f4d60"} Nov 02 15:34:31.769500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.768565 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" event={"ID":"94193325-ecad-4c9d-949b-4c1e83cda24f","Type":"ContainerStarted","Data":"096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96"} Nov 02 15:34:31.771791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.771636 4469 generic.go:356] "Generic (PLEG): container finished" podID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerID="ec01e8a7d3bbc80e82dff2e2f3b051570beedb6504329c03910ea538a7f4c5a7" exitCode=0 Nov 02 15:34:31.771791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.771733 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" event={"ID":"de0528b4-a52b-4f8c-8613-144e25e9c95f","Type":"ContainerDied","Data":"ec01e8a7d3bbc80e82dff2e2f3b051570beedb6504329c03910ea538a7f4c5a7"} Nov 02 15:34:31.843841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.843790 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:34:31.898318 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.898115 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" podStartSLOduration=1.8980998119999999 podStartE2EDuration="1.898099812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:31.896667205 +0000 UTC m=+2274.780269725" watchObservedRunningTime="2026-11-02 15:34:31.898099812 +0000 UTC m=+2274.781702331" Nov 02 15:34:31.912279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.912255 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:34:31.937755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.937729 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\") pod \"de0528b4-a52b-4f8c-8613-144e25e9c95f\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " Nov 02 15:34:31.937898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.937787 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\") pod \"ab4c08af-9592-414b-945e-6497bc057e57\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " Nov 02 15:34:31.937898 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.937877 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\" (UniqueName: \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\") pod \"ab4c08af-9592-414b-945e-6497bc057e57\" (UID: \"ab4c08af-9592-414b-945e-6497bc057e57\") " Nov 02 15:34:31.938017 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.937993 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\" (UniqueName: \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\") pod \"de0528b4-a52b-4f8c-8613-144e25e9c95f\" (UID: \"de0528b4-a52b-4f8c-8613-144e25e9c95f\") " Nov 02 15:34:31.938140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.938114 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume" pod "de0528b4-a52b-4f8c-8613-144e25e9c95f" (UID: "de0528b4-a52b-4f8c-8613-144e25e9c95f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:31.938200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.938163 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume" pod "ab4c08af-9592-414b-945e-6497bc057e57" (UID: "ab4c08af-9592-414b-945e-6497bc057e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:31.938587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.938562 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0528b4-a52b-4f8c-8613-144e25e9c95f-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:31.938713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.938587 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4c08af-9592-414b-945e-6497bc057e57-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:31.940068 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.940045 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69" pod "ab4c08af-9592-414b-945e-6497bc057e57" (UID: "ab4c08af-9592-414b-945e-6497bc057e57"). InnerVolumeSpecName "kube-api-access-vsm69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:31.940155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:31.940081 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9" pod "de0528b4-a52b-4f8c-8613-144e25e9c95f" (UID: "de0528b4-a52b-4f8c-8613-144e25e9c95f"). InnerVolumeSpecName "kube-api-access-hbdz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:32.040037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.039994 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsm69\" (UniqueName: \"kubernetes.io/projected/ab4c08af-9592-414b-945e-6497bc057e57-kube-api-access-vsm69\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:32.040037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.040038 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbdz9\" (UniqueName: \"kubernetes.io/projected/de0528b4-a52b-4f8c-8613-144e25e9c95f-kube-api-access-hbdz9\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:32.779529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.779497 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" Nov 02 15:34:32.779529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.779512 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb" event={"ID":"ab4c08af-9592-414b-945e-6497bc057e57","Type":"ContainerDied","Data":"1c30efd4dd9b0fee30555bd0a95d4ee746b06fecdfaffa19a62c829b0f5ceb42"} Nov 02 15:34:32.780061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.779584 4469 scope.go:122] "RemoveContainer" containerID="4e368d5db8e9a7d49f5a12dd565b74cb6b2899359d2b39c5e3d4919357456394" Nov 02 15:34:32.781330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.781300 4469 generic.go:356] "Generic (PLEG): container finished" podID="94193325-ecad-4c9d-949b-4c1e83cda24f" containerID="6cbe997421e9ec7c5b6a3bb3eb072ba7d06e884d259d542d7e8bbceb949f4d60" exitCode=0 Nov 02 15:34:32.781420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.781397 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" event={"ID":"94193325-ecad-4c9d-949b-4c1e83cda24f","Type":"ContainerDied","Data":"6cbe997421e9ec7c5b6a3bb3eb072ba7d06e884d259d542d7e8bbceb949f4d60"} Nov 02 15:34:32.782695 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.782670 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" event={"ID":"de0528b4-a52b-4f8c-8613-144e25e9c95f","Type":"ContainerDied","Data":"3213a78a0388a9c41b3fc60eea08f8398dc7443e2952c333678cd4a6f3bcba99"} Nov 02 15:34:32.782871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.782684 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6" Nov 02 15:34:32.795736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.795719 4469 scope.go:122] "RemoveContainer" containerID="ec01e8a7d3bbc80e82dff2e2f3b051570beedb6504329c03910ea538a7f4c5a7" Nov 02 15:34:32.946192 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.946101 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:34:32.950860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:32.950826 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-42555d-7c658fbd6c-st4wb"] Nov 02 15:34:33.016103 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.016071 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:33.210260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.210230 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:34:33.215203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.215169 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-42555d-756d9fcdc-mt9h6"] Nov 02 15:34:33.238724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.238699 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4c08af-9592-414b-945e-6497bc057e57" path="/var/lib/kubelet/pods/ab4c08af-9592-414b-945e-6497bc057e57/volumes" Nov 02 15:34:33.239162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.239145 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" path="/var/lib/kubelet/pods/de0528b4-a52b-4f8c-8613-144e25e9c95f/volumes" Nov 02 15:34:33.931353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.931331 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:33.958850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.958825 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\" (UniqueName: \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\") pod \"94193325-ecad-4c9d-949b-4c1e83cda24f\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " Nov 02 15:34:33.958969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.958888 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\") pod \"94193325-ecad-4c9d-949b-4c1e83cda24f\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " Nov 02 15:34:33.959016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.958999 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\") pod \"94193325-ecad-4c9d-949b-4c1e83cda24f\" (UID: \"94193325-ecad-4c9d-949b-4c1e83cda24f\") " Nov 02 15:34:33.959354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.959328 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca" pod "94193325-ecad-4c9d-949b-4c1e83cda24f" (UID: "94193325-ecad-4c9d-949b-4c1e83cda24f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:33.960723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.960701 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca" pod "94193325-ecad-4c9d-949b-4c1e83cda24f" (UID: "94193325-ecad-4c9d-949b-4c1e83cda24f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:33.960841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:33.960831 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd" pod "94193325-ecad-4c9d-949b-4c1e83cda24f" (UID: "94193325-ecad-4c9d-949b-4c1e83cda24f"). InnerVolumeSpecName "kube-api-access-hdqnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:34.060852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.060776 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqnd\" (UniqueName: \"kubernetes.io/projected/94193325-ecad-4c9d-949b-4c1e83cda24f-kube-api-access-hdqnd\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:34.060852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.060834 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:34.060852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.060851 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94193325-ecad-4c9d-949b-4c1e83cda24f-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:34.710994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.710960 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:34:34.711541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711527 4469 cpu_manager.go:414] "containerMap: removing container" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" Nov 02 15:34:34.711591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711543 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" Nov 02 15:34:34.711591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711558 4469 cpu_manager.go:414] "containerMap: removing container" podUID="94193325-ecad-4c9d-949b-4c1e83cda24f" containerName="revoke-keys" Nov 02 15:34:34.711591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711566 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94193325-ecad-4c9d-949b-4c1e83cda24f" containerName="revoke-keys" Nov 02 15:34:34.711699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711593 4469 cpu_manager.go:414] "containerMap: removing container" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" Nov 02 15:34:34.711699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711599 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" Nov 02 15:34:34.711699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711676 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0528b4-a52b-4f8c-8613-144e25e9c95f" containerName="payload-pre-processing" Nov 02 15:34:34.711699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711690 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4c08af-9592-414b-945e-6497bc057e57" containerName="payload-processing" Nov 02 15:34:34.711699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.711700 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="94193325-ecad-4c9d-949b-4c1e83cda24f" containerName="revoke-keys" Nov 02 15:34:34.714712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.714697 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.768634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768597 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.768634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768632 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.768894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768733 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.768894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768770 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.768894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768788 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.769060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.768952 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.769120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.769061 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfgm\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.769182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.769133 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.769182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.769160 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.798560 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.798537 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" Nov 02 15:34:34.798710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.798581 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc" event={"ID":"94193325-ecad-4c9d-949b-4c1e83cda24f","Type":"ContainerDied","Data":"096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96"} Nov 02 15:34:34.798710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.798622 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096af8cdfe9cb40ae8048ff98df6f28a807f4740b2634cee687804a5d2fc7e96" Nov 02 15:34:34.820857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.820833 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:34:34.870013 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.869979 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870072 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870090 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870132 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870160 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870187 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870265 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870343 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfgm\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870380 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870499 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870532 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870568 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870641 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.870980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.870959 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.872464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.872443 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.872651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.872633 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.931418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.931385 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:34.931777 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:34.931521 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfgm\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:35.028356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.028278 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:35.454219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.454190 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:34:35.455760 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:34:35.455735 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fec072_0533_449e_bbe9_a83e656b553f.slice/crio-fb53d940f01085257f8ead58c2450d44736caa92ef051b7610094ae63acc72e2 WatchSource:0}: Error finding container fb53d940f01085257f8ead58c2450d44736caa92ef051b7610094ae63acc72e2: Status 404 returned error can't find the container with id fb53d940f01085257f8ead58c2450d44736caa92ef051b7610094ae63acc72e2 Nov 02 15:34:35.458470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.458438 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:35.458551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.458503 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:35.458587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.458549 4469 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 15:34:35.816258 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.816208 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" event={"ID":"c6fec072-0533-449e-bbe9-a83e656b553f","Type":"ContainerStarted","Data":"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6"} Nov 02 15:34:35.816411 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.816313 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" event={"ID":"c6fec072-0533-449e-bbe9-a83e656b553f","Type":"ContainerStarted","Data":"fb53d940f01085257f8ead58c2450d44736caa92ef051b7610094ae63acc72e2"} Nov 02 15:34:35.950716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:35.950659 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podStartSLOduration=1.950640066 podStartE2EDuration="1.950640066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:35.949019438 +0000 UTC m=+2278.832621958" watchObservedRunningTime="2026-11-02 15:34:35.950640066 +0000 UTC m=+2278.834242588" Nov 02 15:34:36.030650 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:36.029860 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:36.033979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:36.033946 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:36.034153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:36.034026 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:36.189089 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:34:36.188993 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:34:37.039251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:37.039206 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:37.039790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:37.039290 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:37.959561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:37.959525 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:38.040821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:38.029197 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:38.040821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:38.029263 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:38.163370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:38.163311 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:38.470647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:38.470606 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:38.679220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:38.679193 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:39.041825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:39.036197 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:39.041825 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:39.036258 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:40.043235 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:40.043200 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:40.043853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:40.043821 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:41.037824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:41.031241 4469 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:41.037824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:41.031310 4469 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:34:41.393515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:41.393421 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:41.638041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:41.638001 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:42.033389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:42.033359 4469 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:42.033766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:42.033745 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:42.034677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:42.034656 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:34:43.944779 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:43.944723 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:45.079942 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:45.079908 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:45.296561 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:45.296526 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:46.194717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:46.194683 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:46.635527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:46.635486 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:48.344905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.344864 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:48.667602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.667519 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd"] Nov 02 15:34:48.673043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.673018 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.731408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.731373 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd"] Nov 02 15:34:48.770984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.770950 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:34:48.771177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.771022 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:34:48.796628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.796601 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.796791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.796657 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxf7\" (UniqueName: \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.796892 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.796785 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.898101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.898068 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.898254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.898129 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxf7\" (UniqueName: \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.898254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.898174 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.898703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.898683 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.900539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.900516 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.960535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.960503 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxf7\" (UniqueName: \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\") pod \"maas-api-revoke-sub-880f96ce-bz2xd\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:48.988304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:48.988281 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:49.373701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:49.373673 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd"] Nov 02 15:34:49.374673 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:34:49.374639 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec83ff38_a3fa_47e2_b64e_ac01a33d869f.slice/crio-0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4 WatchSource:0}: Error finding container 0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4: Status 404 returned error can't find the container with id 0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4 Nov 02 15:34:49.495757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:49.492394 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:49.925715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:49.925624 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" event={"ID":"ec83ff38-a3fa-47e2-b64e-ac01a33d869f","Type":"ContainerStarted","Data":"606e1b4c5f6cbade88207f9c4bf75c3e8c3ff84252f6e5e7e86cfc109b69d1fa"} Nov 02 15:34:49.925715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:49.925688 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" event={"ID":"ec83ff38-a3fa-47e2-b64e-ac01a33d869f","Type":"ContainerStarted","Data":"0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4"} Nov 02 15:34:50.052741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:50.052663 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" podStartSLOduration=2.052644786 podStartE2EDuration="2.052644786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:50.052007243 +0000 UTC m=+2292.935609764" watchObservedRunningTime="2026-11-02 15:34:50.052644786 +0000 UTC m=+2292.936247306" Nov 02 15:34:50.413200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:50.413162 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:34:50.932915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:50.932872 4469 generic.go:356] "Generic (PLEG): container finished" podID="ec83ff38-a3fa-47e2-b64e-ac01a33d869f" containerID="606e1b4c5f6cbade88207f9c4bf75c3e8c3ff84252f6e5e7e86cfc109b69d1fa" exitCode=0 Nov 02 15:34:50.933102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:50.932923 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" event={"ID":"ec83ff38-a3fa-47e2-b64e-ac01a33d869f","Type":"ContainerDied","Data":"606e1b4c5f6cbade88207f9c4bf75c3e8c3ff84252f6e5e7e86cfc109b69d1fa"} Nov 02 15:34:52.456290 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.455977 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:52.610594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.610562 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\" (UniqueName: \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\") pod \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " Nov 02 15:34:52.610761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.610620 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\") pod \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " Nov 02 15:34:52.610761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.610672 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\") pod \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\" (UID: \"ec83ff38-a3fa-47e2-b64e-ac01a33d869f\") " Nov 02 15:34:52.611268 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.611229 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca" pod "ec83ff38-a3fa-47e2-b64e-ac01a33d869f" (UID: "ec83ff38-a3fa-47e2-b64e-ac01a33d869f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:52.611568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.611541 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.613107 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.613083 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca" pod "ec83ff38-a3fa-47e2-b64e-ac01a33d869f" (UID: "ec83ff38-a3fa-47e2-b64e-ac01a33d869f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:52.615193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.615170 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7" pod "ec83ff38-a3fa-47e2-b64e-ac01a33d869f" (UID: "ec83ff38-a3fa-47e2-b64e-ac01a33d869f"). InnerVolumeSpecName "kube-api-access-xhxf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:52.712493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.712456 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxf7\" (UniqueName: \"kubernetes.io/projected/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-kube-api-access-xhxf7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.712493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:52.712491 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec83ff38-a3fa-47e2-b64e-ac01a33d869f-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:34:53.010404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:53.010319 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" event={"ID":"ec83ff38-a3fa-47e2-b64e-ac01a33d869f","Type":"ContainerDied","Data":"0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4"} Nov 02 15:34:53.010404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:53.010383 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e86009659f13181b02dc131d0604a3f21585be9a65936302b17ac6f3daf16a4" Nov 02 15:34:53.010665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:53.010451 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd" Nov 02 15:34:53.720111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:53.720073 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:34:53.828553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:53.826074 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:34:54.143488 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:54.143350 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:54.393074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:54.393043 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:54.612990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:54.612942 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:55.078994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:55.078952 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:56.181392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:56.181349 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:58.721452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:58.721394 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" containerID="cri-o://8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366" gracePeriod=25 Nov 02 15:34:58.827269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:34:58.827200 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" containerID="cri-o://0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9" gracePeriod=25 Nov 02 15:35:00.153761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:00.153709 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:04.414707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.414671 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg"] Nov 02 15:35:04.415310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.415292 4469 cpu_manager.go:414] "containerMap: removing container" podUID="ec83ff38-a3fa-47e2-b64e-ac01a33d869f" containerName="revoke-keys" Nov 02 15:35:04.415362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.415312 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec83ff38-a3fa-47e2-b64e-ac01a33d869f" containerName="revoke-keys" Nov 02 15:35:04.415418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.415406 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec83ff38-a3fa-47e2-b64e-ac01a33d869f" containerName="revoke-keys" Nov 02 15:35:04.418708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.418691 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.462703 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.462674 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ht7\" (UniqueName: \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.462868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.462848 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.463036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.463015 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.482313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.482286 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg"] Nov 02 15:35:04.520132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.520103 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:35:04.520316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.520213 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:35:04.564352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.564319 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.564502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.564425 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.564502 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.564481 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ht7\" (UniqueName: \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.565021 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.564998 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.566646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.566625 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.627185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.627152 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ht7\" (UniqueName: \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\") pod \"maas-api-revoke-sub-7ceedac8-pd4lg\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:04.635217 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.635187 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:04.732646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:04.732607 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:05.114792 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:05.114717 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg"] Nov 02 15:35:05.119258 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:35:05.119229 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb92fb1_6349_4596_97f7_0aa30b29f562.slice/crio-edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f WatchSource:0}: Error finding container edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f: Status 404 returned error can't find the container with id edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f Nov 02 15:35:06.104249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.104214 4469 generic.go:356] "Generic (PLEG): container finished" podID="aeb92fb1-6349-4596-97f7-0aa30b29f562" containerID="029350defde85bcbfbfe304cbc6470548b76f247bb83184c8896a20a373ec79b" exitCode=0 Nov 02 15:35:06.104574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.104292 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" event={"ID":"aeb92fb1-6349-4596-97f7-0aa30b29f562","Type":"ContainerDied","Data":"029350defde85bcbfbfe304cbc6470548b76f247bb83184c8896a20a373ec79b"} Nov 02 15:35:06.104574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.104349 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" event={"ID":"aeb92fb1-6349-4596-97f7-0aa30b29f562","Type":"ContainerStarted","Data":"edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f"} Nov 02 15:35:06.186013 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:06.185975 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:35:06.300763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.300707 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:06.628187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.627587 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:06.906862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:06.906748 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:07.084718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.084681 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:35:07.084912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.084729 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:35:07.265386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.265362 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:07.317225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.317202 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:07.426133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.426048 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\" (UniqueName: \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\") pod \"aeb92fb1-6349-4596-97f7-0aa30b29f562\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " Nov 02 15:35:07.426303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.426207 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\") pod \"aeb92fb1-6349-4596-97f7-0aa30b29f562\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " Nov 02 15:35:07.426303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.426252 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\") pod \"aeb92fb1-6349-4596-97f7-0aa30b29f562\" (UID: \"aeb92fb1-6349-4596-97f7-0aa30b29f562\") " Nov 02 15:35:07.426663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.426632 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca" pod "aeb92fb1-6349-4596-97f7-0aa30b29f562" (UID: "aeb92fb1-6349-4596-97f7-0aa30b29f562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:07.426831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.426776 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:07.428322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.428293 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7" pod "aeb92fb1-6349-4596-97f7-0aa30b29f562" (UID: "aeb92fb1-6349-4596-97f7-0aa30b29f562"). InnerVolumeSpecName "kube-api-access-b7ht7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:07.428435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.428338 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca" pod "aeb92fb1-6349-4596-97f7-0aa30b29f562" (UID: "aeb92fb1-6349-4596-97f7-0aa30b29f562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:07.527969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.527942 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ht7\" (UniqueName: \"kubernetes.io/projected/aeb92fb1-6349-4596-97f7-0aa30b29f562-kube-api-access-b7ht7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:07.527969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:07.527968 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb92fb1-6349-4596-97f7-0aa30b29f562-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:08.116090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:08.116058 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" Nov 02 15:35:08.116270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:08.116095 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg" event={"ID":"aeb92fb1-6349-4596-97f7-0aa30b29f562","Type":"ContainerDied","Data":"edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f"} Nov 02 15:35:08.116270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:08.116142 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edcdf39ef094e481a7c7b2b24782b79971313055c239f920ec1bf69e263c977f" Nov 02 15:35:09.392420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:09.392362 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:10.041169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:10.041138 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:11.960469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:11.960427 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:12.355690 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:12.355600 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:16.890700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:16.890662 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:17.084720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:17.084684 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:35:17.084720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:17.084698 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:35:19.013656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.013628 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:35:19.080024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.080000 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\") pod \"929a0590-726f-488c-9af7-7c4d55d554fb\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " Nov 02 15:35:19.080245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.080227 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\" (UniqueName: \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\") pod \"929a0590-726f-488c-9af7-7c4d55d554fb\" (UID: \"929a0590-726f-488c-9af7-7c4d55d554fb\") " Nov 02 15:35:19.080434 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.080410 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume" pod "929a0590-726f-488c-9af7-7c4d55d554fb" (UID: "929a0590-726f-488c-9af7-7c4d55d554fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:19.080710 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.080689 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929a0590-726f-488c-9af7-7c4d55d554fb-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:19.082269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.082249 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl" pod "929a0590-726f-488c-9af7-7c4d55d554fb" (UID: "929a0590-726f-488c-9af7-7c4d55d554fb"). InnerVolumeSpecName "kube-api-access-bhlrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:19.090223 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.090204 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:35:19.181206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.181174 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\" (UniqueName: \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\") pod \"7df298a8-465a-41e9-9902-c01eccae7da5\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " Nov 02 15:35:19.181397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.181271 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\") pod \"7df298a8-465a-41e9-9902-c01eccae7da5\" (UID: \"7df298a8-465a-41e9-9902-c01eccae7da5\") " Nov 02 15:35:19.181698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.181658 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume" pod "7df298a8-465a-41e9-9902-c01eccae7da5" (UID: "7df298a8-465a-41e9-9902-c01eccae7da5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:19.181947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.181923 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlrl\" (UniqueName: \"kubernetes.io/projected/929a0590-726f-488c-9af7-7c4d55d554fb-kube-api-access-bhlrl\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:19.182054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.181954 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df298a8-465a-41e9-9902-c01eccae7da5-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:19.183567 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.183542 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz" pod "7df298a8-465a-41e9-9902-c01eccae7da5" (UID: "7df298a8-465a-41e9-9902-c01eccae7da5"). InnerVolumeSpecName "kube-api-access-szxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:19.187527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.187505 4469 generic.go:356] "Generic (PLEG): container finished" podID="7df298a8-465a-41e9-9902-c01eccae7da5" containerID="0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9" exitCode=0 Nov 02 15:35:19.187628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.187559 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" Nov 02 15:35:19.187628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.187582 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" event={"ID":"7df298a8-465a-41e9-9902-c01eccae7da5","Type":"ContainerDied","Data":"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9"} Nov 02 15:35:19.187746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.187634 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45" event={"ID":"7df298a8-465a-41e9-9902-c01eccae7da5","Type":"ContainerDied","Data":"da3af2477e56c42bc5a9cdc9aa9356446022bf1984ef468685716d746c92fb37"} Nov 02 15:35:19.187746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.187657 4469 scope.go:122] "RemoveContainer" containerID="0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9" Nov 02 15:35:19.189158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.189130 4469 generic.go:356] "Generic (PLEG): container finished" podID="929a0590-726f-488c-9af7-7c4d55d554fb" containerID="8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366" exitCode=0 Nov 02 15:35:19.189262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.189171 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" Nov 02 15:35:19.189262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.189207 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" event={"ID":"929a0590-726f-488c-9af7-7c4d55d554fb","Type":"ContainerDied","Data":"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366"} Nov 02 15:35:19.189262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.189255 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn" event={"ID":"929a0590-726f-488c-9af7-7c4d55d554fb","Type":"ContainerDied","Data":"2d109047fe7d562ed47ff84bebd935a73f4a1cb0bb7c5fe41244d77bcecd09f8"} Nov 02 15:35:19.198534 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.198515 4469 scope.go:122] "RemoveContainer" containerID="0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9" Nov 02 15:35:19.198787 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:19.198768 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9\": container with ID starting with 0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9 not found: ID does not exist" containerID="0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9" Nov 02 15:35:19.198905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.198791 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9"} err="failed to get container status \"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9\": rpc error: code = NotFound desc = could not find container \"0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9\": container with ID starting with 0a9e3b6b7d4c3c69007553b5e9d63351d2ae3a7106432bbd613cac3b2eb83bf9 not found: ID does not exist" Nov 02 15:35:19.198905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.198828 4469 scope.go:122] "RemoveContainer" containerID="8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366" Nov 02 15:35:19.207112 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.207092 4469 scope.go:122] "RemoveContainer" containerID="8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366" Nov 02 15:35:19.207362 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:19.207343 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366\": container with ID starting with 8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366 not found: ID does not exist" containerID="8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366" Nov 02 15:35:19.207425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.207373 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366"} err="failed to get container status \"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366\": rpc error: code = NotFound desc = could not find container \"8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366\": container with ID starting with 8042c7c2495853e083321032becf0a782f2449b1b9d125f19692aeea10591366 not found: ID does not exist" Nov 02 15:35:19.283080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.282997 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxhz\" (UniqueName: \"kubernetes.io/projected/7df298a8-465a-41e9-9902-c01eccae7da5-kube-api-access-szxhz\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:19.365205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.365173 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:35:19.368702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.368675 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-de35cd-7dddfd879-8kn45"] Nov 02 15:35:19.484708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.484675 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:35:19.490238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:19.490212 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de35cd-c8f875df5-fj5qn"] Nov 02 15:35:21.239213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:21.239175 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" path="/var/lib/kubelet/pods/7df298a8-465a-41e9-9902-c01eccae7da5/volumes" Nov 02 15:35:21.239603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:21.239589 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" path="/var/lib/kubelet/pods/929a0590-726f-488c-9af7-7c4d55d554fb/volumes" Nov 02 15:35:21.866407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:21.866361 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:23.096064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:23.094102 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:23.312981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:23.312950 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:25.492769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:25.492735 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:26.180930 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:26.180890 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:26.802863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:26.802829 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:31.093057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:31.093021 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:32.053954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:32.053917 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:32.730617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:32.730581 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:32.952109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:32.952078 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:33.169647 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:33.169551 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:33.664712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:33.664675 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:36.187277 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:36.187227 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:35:38.091093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:38.091046 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:40.250397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.250360 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:40.466031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.465992 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:40.498850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.498815 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:35:40.499054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.499029 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" containerID="cri-o://c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec" gracePeriod=30 Nov 02 15:35:40.515312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.515254 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:35:40.515462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.515438 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" containerID="cri-o://cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f" gracePeriod=30 Nov 02 15:35:40.719011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:40.718978 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:43.037583 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:43.037538 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:44.246619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:44.246584 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:35:45.077791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.077744 4469 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:35:45.078314 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.078286 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:35:45.780165 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.780095 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:35:45.815028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.815004 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:35:45.870452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870418 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870463 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870542 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870616 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870663 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870723 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870767 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870840 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.870854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870839 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.871079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870879 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\") pod \"7200a789-6164-4827-b0e2-d5ba74319a92\" (UID: \"7200a789-6164-4827-b0e2-d5ba74319a92\") " Nov 02 15:35:45.871079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870896 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.871079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.870912 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:45.871079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871028 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.871309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871102 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.871582 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871554 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.871679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871589 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7200a789-6164-4827-b0e2-d5ba74319a92-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.871679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871606 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.871679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871623 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.871679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.871637 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.872834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.872782 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.873149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.873133 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:45.873353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.873328 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:35:45.873353 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.873347 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7" pod "7200a789-6164-4827-b0e2-d5ba74319a92" (UID: "7200a789-6164-4827-b0e2-d5ba74319a92"). InnerVolumeSpecName "kube-api-access-kvck7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:45.972585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972542 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972588 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972645 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972687 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972727 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972825 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.972865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972862 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.973151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972874 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.973151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972920 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.973151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.972976 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.973151 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973034 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\") pod \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\" (UID: \"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1\") " Nov 02 15:35:45.973379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973274 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:45.973443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973376 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973623 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973704 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973723 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7200a789-6164-4827-b0e2-d5ba74319a92-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973726 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973736 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973772 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973791 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvck7\" (UniqueName: \"kubernetes.io/projected/7200a789-6164-4827-b0e2-d5ba74319a92-kube-api-access-kvck7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973833 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.973865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.973849 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7200a789-6164-4827-b0e2-d5ba74319a92-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:45.975132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.975071 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:45.975261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.975217 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "kube-api-access-22cgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:45.975581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.975563 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:45.975675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:45.975592 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo" pod "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" (UID: "f8ce43d8-a349-4af9-a379-d4c4cf4d06b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:35:46.074600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.074519 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:46.074600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.074551 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:46.074600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.074564 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cgj\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-kube-api-access-22cgj\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:46.074600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.074578 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:46.074600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.074591 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:35:46.348500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.348412 4469 generic.go:356] "Generic (PLEG): container finished" podID="7200a789-6164-4827-b0e2-d5ba74319a92" containerID="c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec" exitCode=0 Nov 02 15:35:46.348500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.348469 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" Nov 02 15:35:46.348721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.348495 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" event={"ID":"7200a789-6164-4827-b0e2-d5ba74319a92","Type":"ContainerDied","Data":"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec"} Nov 02 15:35:46.348721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.348547 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc" event={"ID":"7200a789-6164-4827-b0e2-d5ba74319a92","Type":"ContainerDied","Data":"6ded0c0261e49bd919b12271fb5b8fa525af816ed534e2b15e1a7f8b73f57532"} Nov 02 15:35:46.348721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.348573 4469 scope.go:122] "RemoveContainer" containerID="c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec" Nov 02 15:35:46.350048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.350023 4469 generic.go:356] "Generic (PLEG): container finished" podID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerID="cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f" exitCode=0 Nov 02 15:35:46.350180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.350062 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" Nov 02 15:35:46.350180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.350081 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" event={"ID":"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1","Type":"ContainerDied","Data":"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f"} Nov 02 15:35:46.350180 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.350123 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn" event={"ID":"f8ce43d8-a349-4af9-a379-d4c4cf4d06b1","Type":"ContainerDied","Data":"353d1f23b3c9bcb9a04ddccfd0ca5b0a550b4401ff57905724b10ebd0a4a5a21"} Nov 02 15:35:46.360132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.360110 4469 scope.go:122] "RemoveContainer" containerID="c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec" Nov 02 15:35:46.360433 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:46.360414 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec\": container with ID starting with c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec not found: ID does not exist" containerID="c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec" Nov 02 15:35:46.360600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.360443 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec"} err="failed to get container status \"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec\": rpc error: code = NotFound desc = could not find container \"c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec\": container with ID starting with c8765ed34e0eddcf1f3bb20c5e4df6e9aea2b6912142214c8196cad73e56deec not found: ID does not exist" Nov 02 15:35:46.360600 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.360465 4469 scope.go:122] "RemoveContainer" containerID="cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f" Nov 02 15:35:46.369152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.369134 4469 scope.go:122] "RemoveContainer" containerID="cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f" Nov 02 15:35:46.369402 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:35:46.369385 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f\": container with ID starting with cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f not found: ID does not exist" containerID="cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f" Nov 02 15:35:46.369458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.369414 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f"} err="failed to get container status \"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f\": rpc error: code = NotFound desc = could not find container \"cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f\": container with ID starting with cead7515b567ff233675fe1b970e46e28d2b923f2cca38e706a8236bee330f0f not found: ID does not exist" Nov 02 15:35:46.517836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.517790 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:35:46.544824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.544772 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:46.577782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.577748 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-bq2wn"] Nov 02 15:35:46.638396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.638321 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:35:46.645337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:46.645313 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zntdc"] Nov 02 15:35:47.241044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:47.241014 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" path="/var/lib/kubelet/pods/7200a789-6164-4827-b0e2-d5ba74319a92/volumes" Nov 02 15:35:47.241538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:47.241521 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" path="/var/lib/kubelet/pods/f8ce43d8-a349-4af9-a379-d4c4cf4d06b1/volumes" Nov 02 15:35:47.336359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:47.336318 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:48.562743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:48.562714 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:48.891418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:48.891332 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:49.145507 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:49.145436 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:49.380005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:49.379968 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:49.884016 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:49.883982 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:50.653225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:50.653176 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:54.560156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:54.560119 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:56.457095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:56.457063 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:56.675506 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:56.675446 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:56.691110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:56.691078 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:35:56.691338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:56.691315 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" containerID="cri-o://89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62" gracePeriod=30 Nov 02 15:35:56.895759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:56.895722 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:57.715563 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:57.715498 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:58.818259 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:58.818221 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:59.457755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:35:59.457713 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:01.954108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:01.954085 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:36:02.082142 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082111 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082210 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082273 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082299 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082337 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082374 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082415 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:02.082599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082544 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082641 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.082763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082651 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:02.082763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082678 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:02.082763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.082689 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\") pod \"73e95c7e-2901-4a66-aa55-66c0587cab08\" (UID: \"73e95c7e-2901-4a66-aa55-66c0587cab08\") " Nov 02 15:36:02.083141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083102 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:02.083369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083348 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.083424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083379 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.083424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083395 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.083424 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083409 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.083642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.083623 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:02.084641 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.084617 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "kube-api-access-d6ksx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:02.084727 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.084688 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:02.084891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.084867 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:02.084960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.084919 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo" pod "73e95c7e-2901-4a66-aa55-66c0587cab08" (UID: "73e95c7e-2901-4a66-aa55-66c0587cab08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:02.185297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.185081 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.185297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.185115 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ksx\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-kube-api-access-d6ksx\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.185297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.185128 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.185297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.185141 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e95c7e-2901-4a66-aa55-66c0587cab08-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.185297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.185154 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e95c7e-2901-4a66-aa55-66c0587cab08-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:02.448527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.448428 4469 generic.go:356] "Generic (PLEG): container finished" podID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerID="89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62" exitCode=0 Nov 02 15:36:02.448527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.448500 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" Nov 02 15:36:02.448735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.448555 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerDied","Data":"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62"} Nov 02 15:36:02.448735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.448610 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb" event={"ID":"73e95c7e-2901-4a66-aa55-66c0587cab08","Type":"ContainerDied","Data":"37d059c703106a347066889f48b82ff39ce374434198d7ea53acd6da1abe7313"} Nov 02 15:36:02.448735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.448640 4469 scope.go:122] "RemoveContainer" containerID="89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62" Nov 02 15:36:02.459219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.459201 4469 scope.go:122] "RemoveContainer" containerID="854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f" Nov 02 15:36:02.467362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.467346 4469 scope.go:122] "RemoveContainer" containerID="89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62" Nov 02 15:36:02.467628 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:02.467610 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62\": container with ID starting with 89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62 not found: ID does not exist" containerID="89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62" Nov 02 15:36:02.467705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.467641 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62"} err="failed to get container status \"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62\": rpc error: code = NotFound desc = could not find container \"89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62\": container with ID starting with 89d25e5cafe0e09a2aec4b268eedbbc468d92c5744fbcf3df580d886c47f3e62 not found: ID does not exist" Nov 02 15:36:02.467705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.467665 4469 scope.go:122] "RemoveContainer" containerID="854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f" Nov 02 15:36:02.467938 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:02.467919 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f\": container with ID starting with 854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f not found: ID does not exist" containerID="854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f" Nov 02 15:36:02.467983 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.467945 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f"} err="failed to get container status \"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f\": rpc error: code = NotFound desc = could not find container \"854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f\": container with ID starting with 854bb8f1973becbf02a3fc30b30b0c5605e51131df35d75d4eb6dada41b1930f not found: ID does not exist" Nov 02 15:36:02.614765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.614727 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:36:02.664835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:02.664791 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-tlwnb"] Nov 02 15:36:03.238658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:03.238628 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" path="/var/lib/kubelet/pods/73e95c7e-2901-4a66-aa55-66c0587cab08/volumes" Nov 02 15:36:04.476136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:04.476095 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:06.185465 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:06.185432 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:36:07.850023 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:07.849990 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:08.093106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:08.093072 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:09.441499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:09.441460 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:10.405817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:10.405766 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:11.286598 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:11.286566 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:36:11.542369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:11.542287 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:14.639549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.639519 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:14.859852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.859779 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:14.925789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.925716 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:36:14.925988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.925963 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" containerID="cri-o://b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e" gracePeriod=30 Nov 02 15:36:14.952176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.952142 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:36:14.952361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.952340 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" containerID="cri-o://8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6" gracePeriod=30 Nov 02 15:36:14.974503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.974473 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:36:14.974742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:14.974694 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" containerID="cri-o://09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97" gracePeriod=30 Nov 02 15:36:15.140366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:15.140332 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:15.858669 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:15.858623 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:16.092473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:16.092430 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:16.431208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:16.431176 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:36:16.669693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:16.669657 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:19.768949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:19.768860 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:20.263941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.263903 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:36:20.310293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.310240 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:36:20.313360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.313334 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:36:20.400365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400328 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400558 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400407 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400558 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400467 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400558 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400524 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400575 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400595 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.400755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400707 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400757 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400702 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.400952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400788 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.400952 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.400870 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\") pod \"c6fec072-0533-449e-bbe9-a83e656b553f\" (UID: \"c6fec072-0533-449e-bbe9-a83e656b553f\") " Nov 02 15:36:20.401244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401206 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.401333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401304 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.401836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401511 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.401836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401537 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.401836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401553 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.401836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401567 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.401836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.401575 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:20.403402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.403363 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:20.403402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.403384 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "kube-api-access-xmfgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.403772 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.403751 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.403862 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.403837 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token" pod "c6fec072-0533-449e-bbe9-a83e656b553f" (UID: "c6fec072-0533-449e-bbe9-a83e656b553f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.502171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502140 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.502349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502195 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502242 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502267 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502298 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502327 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502371 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502429 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502477 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502510 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502497 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502527 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502550 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502560 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502607 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.502622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502611 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502665 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502731 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502837 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502851 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502877 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\") pod \"8babd2ea-5857-429f-8748-d75181d42d86\" (UID: \"8babd2ea-5857-429f-8748-d75181d42d86\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502922 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.502959 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\") pod \"28912ade-3ae0-4421-b891-da992963b763\" (UID: \"28912ade-3ae0-4421-b891-da992963b763\") " Nov 02 15:36:20.503186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503166 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:20.503579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503499 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503522 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503536 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fec072-0533-449e-bbe9-a83e656b553f-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503551 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503565 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503579 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fec072-0533-449e-bbe9-a83e656b553f-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503593 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503608 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fec072-0533-449e-bbe9-a83e656b553f-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503621 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503635 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28912ade-3ae0-4421-b891-da992963b763-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.503863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.503649 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfgm\" (UniqueName: \"kubernetes.io/projected/c6fec072-0533-449e-bbe9-a83e656b553f-kube-api-access-xmfgm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.504542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.504154 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.504542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.504280 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.504542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.504284 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.504542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.504369 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:20.505214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505178 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "kube-api-access-qwxkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.505539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505484 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:20.505539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505491 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.505763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505589 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.505763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505674 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.505763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.505676 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:20.506084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.506055 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76" pod "8babd2ea-5857-429f-8748-d75181d42d86" (UID: "8babd2ea-5857-429f-8748-d75181d42d86"). InnerVolumeSpecName "kube-api-access-8zm76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:20.506523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.506505 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo" pod "28912ade-3ae0-4421-b891-da992963b763" (UID: "28912ade-3ae0-4421-b891-da992963b763"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:20.554867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.554789 4469 generic.go:356] "Generic (PLEG): container finished" podID="c6fec072-0533-449e-bbe9-a83e656b553f" containerID="8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6" exitCode=0 Nov 02 15:36:20.554867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.554859 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" Nov 02 15:36:20.555060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.554890 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" event={"ID":"c6fec072-0533-449e-bbe9-a83e656b553f","Type":"ContainerDied","Data":"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6"} Nov 02 15:36:20.555060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.554960 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j" event={"ID":"c6fec072-0533-449e-bbe9-a83e656b553f","Type":"ContainerDied","Data":"fb53d940f01085257f8ead58c2450d44736caa92ef051b7610094ae63acc72e2"} Nov 02 15:36:20.555060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.554991 4469 scope.go:122] "RemoveContainer" containerID="8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6" Nov 02 15:36:20.559515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.559487 4469 generic.go:356] "Generic (PLEG): container finished" podID="8babd2ea-5857-429f-8748-d75181d42d86" containerID="b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e" exitCode=0 Nov 02 15:36:20.559670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.559645 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" Nov 02 15:36:20.560113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.560087 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" event={"ID":"8babd2ea-5857-429f-8748-d75181d42d86","Type":"ContainerDied","Data":"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e"} Nov 02 15:36:20.560210 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.560146 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw" event={"ID":"8babd2ea-5857-429f-8748-d75181d42d86","Type":"ContainerDied","Data":"ad81836cd52432b8c5d266c6c3f9c3977d87c559dcbf7f5011c7fe71f6b0c822"} Nov 02 15:36:20.570191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.564122 4469 generic.go:356] "Generic (PLEG): container finished" podID="28912ade-3ae0-4421-b891-da992963b763" containerID="09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97" exitCode=0 Nov 02 15:36:20.570191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.564198 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerDied","Data":"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97"} Nov 02 15:36:20.570191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.564226 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" Nov 02 15:36:20.570191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.564242 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk" event={"ID":"28912ade-3ae0-4421-b891-da992963b763","Type":"ContainerDied","Data":"30e2cc26d74a23058daf8e9ae3ec4d4d2732953102086cf9d11d612629a70108"} Nov 02 15:36:20.570191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.569977 4469 scope.go:122] "RemoveContainer" containerID="8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6" Nov 02 15:36:20.574697 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:20.573128 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6\": container with ID starting with 8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6 not found: ID does not exist" containerID="8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6" Nov 02 15:36:20.574697 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.573164 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6"} err="failed to get container status \"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6\": rpc error: code = NotFound desc = could not find container \"8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6\": container with ID starting with 8eee9085ff89bc5d82ffe36e63e336c72d4e421f512a4c97dcd64caf48159db6 not found: ID does not exist" Nov 02 15:36:20.574697 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.573187 4469 scope.go:122] "RemoveContainer" containerID="b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e" Nov 02 15:36:20.591394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.591374 4469 scope.go:122] "RemoveContainer" containerID="b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e" Nov 02 15:36:20.591779 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:20.591742 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e\": container with ID starting with b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e not found: ID does not exist" containerID="b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e" Nov 02 15:36:20.591909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.591785 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e"} err="failed to get container status \"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e\": rpc error: code = NotFound desc = could not find container \"b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e\": container with ID starting with b062940a83c9dd85b36df36b6c960532cfefd8ebde9c55eed7f321d33d20d43e not found: ID does not exist" Nov 02 15:36:20.591909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.591822 4469 scope.go:122] "RemoveContainer" containerID="09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97" Nov 02 15:36:20.600544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.600523 4469 scope.go:122] "RemoveContainer" containerID="9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e" Nov 02 15:36:20.605517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605487 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28912ade-3ae0-4421-b891-da992963b763-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.605618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605520 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxkd\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-kube-api-access-qwxkd\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.605618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605539 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.605618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605555 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zm76\" (UniqueName: \"kubernetes.io/projected/8babd2ea-5857-429f-8748-d75181d42d86-kube-api-access-8zm76\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.605618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605570 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8babd2ea-5857-429f-8748-d75181d42d86-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.605618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605586 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605624 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605640 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8babd2ea-5857-429f-8748-d75181d42d86-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605653 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8babd2ea-5857-429f-8748-d75181d42d86-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605667 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605722 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28912ade-3ae0-4421-b891-da992963b763-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.606048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.605738 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28912ade-3ae0-4421-b891-da992963b763-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:20.609471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.609448 4469 scope.go:122] "RemoveContainer" containerID="09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97" Nov 02 15:36:20.609838 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:20.609795 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97\": container with ID starting with 09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97 not found: ID does not exist" containerID="09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97" Nov 02 15:36:20.609913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.609849 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97"} err="failed to get container status \"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97\": rpc error: code = NotFound desc = could not find container \"09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97\": container with ID starting with 09fc8459e89e0b063c170abf3463064f65b75a694975c176b36afefc92c04f97 not found: ID does not exist" Nov 02 15:36:20.609913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.609872 4469 scope.go:122] "RemoveContainer" containerID="9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e" Nov 02 15:36:20.610186 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:20.610162 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e\": container with ID starting with 9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e not found: ID does not exist" containerID="9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e" Nov 02 15:36:20.610273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.610192 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e"} err="failed to get container status \"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e\": rpc error: code = NotFound desc = could not find container \"9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e\": container with ID starting with 9432adc947d0c17a8925f1d4572a88a0203e64d71407b70974205b75de0bad1e not found: ID does not exist" Nov 02 15:36:20.808352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.808316 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:36:20.813421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.813398 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fj6qk"] Nov 02 15:36:20.923968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.923896 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:36:20.933742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:20.933705 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-fszfw"] Nov 02 15:36:21.042761 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:21.042722 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:36:21.048788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:21.048755 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-j2s2j"] Nov 02 15:36:21.238390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:21.238359 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28912ade-3ae0-4421-b891-da992963b763" path="/var/lib/kubelet/pods/28912ade-3ae0-4421-b891-da992963b763/volumes" Nov 02 15:36:21.238856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:21.238830 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8babd2ea-5857-429f-8748-d75181d42d86" path="/var/lib/kubelet/pods/8babd2ea-5857-429f-8748-d75181d42d86/volumes" Nov 02 15:36:21.239300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:21.239286 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" path="/var/lib/kubelet/pods/c6fec072-0533-449e-bbe9-a83e656b553f/volumes" Nov 02 15:36:22.443585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:22.443554 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:36:22.552496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:22.552459 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:36:22.881114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:22.881065 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:23.121775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:23.121740 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:24.775931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:24.775875 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:27.447824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:27.446867 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" containerID="cri-o://5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1" gracePeriod=25 Nov 02 15:36:27.556537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:27.556497 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" containerID="cri-o://9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3" gracePeriod=25 Nov 02 15:36:29.805470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:29.805423 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:31.452663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.452556 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" containerID="cri-o://4d71b0c5af18def826f094b749838fcf20ec0ee52f963b1d7db74550970f5a43" gracePeriod=45 Nov 02 15:36:31.638064 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.638034 4469 generic.go:356] "Generic (PLEG): container finished" podID="db9b2560-ada3-487f-9a7b-0775f9100475" containerID="4d71b0c5af18def826f094b749838fcf20ec0ee52f963b1d7db74550970f5a43" exitCode=0 Nov 02 15:36:31.638227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.638110 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" event={"ID":"db9b2560-ada3-487f-9a7b-0775f9100475","Type":"ContainerDied","Data":"4d71b0c5af18def826f094b749838fcf20ec0ee52f963b1d7db74550970f5a43"} Nov 02 15:36:31.715865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.715840 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:36:31.746489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746458 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.746629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746518 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\" (UniqueName: \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.746629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746538 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.746629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746602 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.746629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746626 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.746926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746879 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\" (UniqueName: \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\") pod \"db9b2560-ada3-487f-9a7b-0775f9100475\" (UID: \"db9b2560-ada3-487f-9a7b-0775f9100475\") " Nov 02 15:36:31.747022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746901 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:31.747022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746911 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:31.747022 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.746919 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:31.747419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.747394 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:31.747490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.747424 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:31.747490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.747440 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:31.748903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.748866 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:31.748991 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.748932 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "kube-api-access-zl9t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:31.748991 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.748961 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm" pod "db9b2560-ada3-487f-9a7b-0775f9100475" (UID: "db9b2560-ada3-487f-9a7b-0775f9100475"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:31.848681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.848652 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9b2560-ada3-487f-9a7b-0775f9100475-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:31.848681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.848677 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db9b2560-ada3-487f-9a7b-0775f9100475-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:31.848681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:31.848688 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9t2\" (UniqueName: \"kubernetes.io/projected/db9b2560-ada3-487f-9a7b-0775f9100475-kube-api-access-zl9t2\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:32.111036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.110956 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:36:32.111036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.110969 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:36:32.646455 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.646421 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" Nov 02 15:36:32.646973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.646460 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh" event={"ID":"db9b2560-ada3-487f-9a7b-0775f9100475","Type":"ContainerDied","Data":"2c8fb71c7150bada9a365c4d7223dbe4bae3429425e32fe1df77b66a7e62f4c7"} Nov 02 15:36:32.646973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.646519 4469 scope.go:122] "RemoveContainer" containerID="4d71b0c5af18def826f094b749838fcf20ec0ee52f963b1d7db74550970f5a43" Nov 02 15:36:32.813209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.813172 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:36:32.862410 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:32.862376 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-2-w5-db3931-kserve-7845cdbb89-cwzgh"] Nov 02 15:36:33.237833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:33.237786 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" path="/var/lib/kubelet/pods/db9b2560-ada3-487f-9a7b-0775f9100475/volumes" Nov 02 15:36:34.848994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:34.848958 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:36.186431 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:36.186392 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:36:37.359987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:37.359949 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:39.183324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183293 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9"] Nov 02 15:36:39.183758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183745 4469 cpu_manager.go:414] "containerMap: removing container" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183759 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183771 4469 cpu_manager.go:414] "containerMap: removing container" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183777 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183786 4469 cpu_manager.go:414] "containerMap: removing container" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183792 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183812 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183817 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183831 4469 cpu_manager.go:414] "containerMap: removing container" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" Nov 02 15:36:39.183841 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183837 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183852 4469 cpu_manager.go:414] "containerMap: removing container" podUID="aeb92fb1-6349-4596-97f7-0aa30b29f562" containerName="revoke-keys" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183857 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb92fb1-6349-4596-97f7-0aa30b29f562" containerName="revoke-keys" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183866 4469 cpu_manager.go:414] "containerMap: removing container" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183871 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183877 4469 cpu_manager.go:414] "containerMap: removing container" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183881 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183901 4469 cpu_manager.go:414] "containerMap: removing container" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183905 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183912 4469 cpu_manager.go:414] "containerMap: removing container" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183916 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183922 4469 cpu_manager.go:414] "containerMap: removing container" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183926 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183989 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.183995 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184002 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e95c7e-2901-4a66-aa55-66c0587cab08" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184010 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="929a0590-726f-488c-9af7-7c4d55d554fb" containerName="payload-processing" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184016 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9b2560-ada3-487f-9a7b-0775f9100475" containerName="main" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184021 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184026 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df298a8-465a-41e9-9902-c01eccae7da5" containerName="payload-pre-processing" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184031 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fec072-0533-449e-bbe9-a83e656b553f" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184039 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ce43d8-a349-4af9-a379-d4c4cf4d06b1" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184046 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb92fb1-6349-4596-97f7-0aa30b29f562" containerName="revoke-keys" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184051 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="8babd2ea-5857-429f-8748-d75181d42d86" containerName="istio-proxy" Nov 02 15:36:39.184129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.184057 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="7200a789-6164-4827-b0e2-d5ba74319a92" containerName="istio-proxy" Nov 02 15:36:39.188625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.188606 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.292688 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.292648 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:36:39.295107 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.295084 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9"] Nov 02 15:36:39.296179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.296157 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:36:39.336449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.336423 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.336573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.336459 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.336573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.336541 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtx5\" (UniqueName: \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.437923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.437834 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.437923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.437879 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.437923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.437920 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtx5\" (UniqueName: \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.438670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.438644 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.440169 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.440148 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.499503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.499469 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtx5\" (UniqueName: \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\") pod \"maas-api-revoke-sub-fe9623aa-6vfg9\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.502346 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.502322 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:39.885349 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.884937 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9"] Nov 02 15:36:39.885349 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:36:39.885230 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2fba0b_1a94_4eba_a6c5_157939eeb911.slice/crio-2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b WatchSource:0}: Error finding container 2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b: Status 404 returned error can't find the container with id 2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b Nov 02 15:36:39.964905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:39.964700 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:40.693971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:40.693934 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" event={"ID":"7d2fba0b-1a94-4eba-a6c5-157939eeb911","Type":"ContainerStarted","Data":"12252660dbb2f4dc2dc08994fef50e46f12a3b1c35189628926f77bc5e4a89e0"} Nov 02 15:36:40.694342 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:40.693992 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" event={"ID":"7d2fba0b-1a94-4eba-a6c5-157939eeb911","Type":"ContainerStarted","Data":"2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b"} Nov 02 15:36:40.812172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:40.812128 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" podStartSLOduration=1.812112415 podStartE2EDuration="1.812112415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:40.809637723 +0000 UTC m=+2403.693240245" watchObservedRunningTime="2026-11-02 15:36:40.812112415 +0000 UTC m=+2403.695714933" Nov 02 15:36:41.700718 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:41.700681 4469 generic.go:356] "Generic (PLEG): container finished" podID="7d2fba0b-1a94-4eba-a6c5-157939eeb911" containerID="12252660dbb2f4dc2dc08994fef50e46f12a3b1c35189628926f77bc5e4a89e0" exitCode=0 Nov 02 15:36:41.701122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:41.700769 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" event={"ID":"7d2fba0b-1a94-4eba-a6c5-157939eeb911","Type":"ContainerDied","Data":"12252660dbb2f4dc2dc08994fef50e46f12a3b1c35189628926f77bc5e4a89e0"} Nov 02 15:36:42.111832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.111781 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:36:42.111998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.111898 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:36:42.855922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.855896 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:42.998225 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.998196 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\") pod \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " Nov 02 15:36:42.998387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.998302 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\" (UniqueName: \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\") pod \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " Nov 02 15:36:42.998434 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.998385 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\") pod \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\" (UID: \"7d2fba0b-1a94-4eba-a6c5-157939eeb911\") " Nov 02 15:36:42.998766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:42.998741 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca" pod "7d2fba0b-1a94-4eba-a6c5-157939eeb911" (UID: "7d2fba0b-1a94-4eba-a6c5-157939eeb911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:43.000400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.000370 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca" pod "7d2fba0b-1a94-4eba-a6c5-157939eeb911" (UID: "7d2fba0b-1a94-4eba-a6c5-157939eeb911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:43.000601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.000573 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5" pod "7d2fba0b-1a94-4eba-a6c5-157939eeb911" (UID: "7d2fba0b-1a94-4eba-a6c5-157939eeb911"). InnerVolumeSpecName "kube-api-access-jwtx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:43.099954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.099927 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:43.099954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.099951 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2fba0b-1a94-4eba-a6c5-157939eeb911-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:43.100115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.099961 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtx5\" (UniqueName: \"kubernetes.io/projected/7d2fba0b-1a94-4eba-a6c5-157939eeb911-kube-api-access-jwtx5\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:43.712691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.712661 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" Nov 02 15:36:43.712691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.712679 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9" event={"ID":"7d2fba0b-1a94-4eba-a6c5-157939eeb911","Type":"ContainerDied","Data":"2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b"} Nov 02 15:36:43.712940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:43.712737 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b494507a602c93d477c23e2d98a93ec8fba1c9f96ee199d62d84d6a12e70f1b" Nov 02 15:36:44.781167 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:44.781125 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:46.905906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:46.905872 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:47.691981 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.691948 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:36:47.731015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.730993 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:36:47.736594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.736569 4469 generic.go:356] "Generic (PLEG): container finished" podID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerID="9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3" exitCode=0 Nov 02 15:36:47.736749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.736615 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" Nov 02 15:36:47.736749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.736651 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" event={"ID":"9fd1784a-b0ba-490d-83c5-3171a145cbd5","Type":"ContainerDied","Data":"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3"} Nov 02 15:36:47.736749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.736703 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx" event={"ID":"9fd1784a-b0ba-490d-83c5-3171a145cbd5","Type":"ContainerDied","Data":"427ce8d20e6af38135573ffdb4bf73c445e92c7f2d4d0940e6f2f6c2d8dd04db"} Nov 02 15:36:47.736749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.736733 4469 scope.go:122] "RemoveContainer" containerID="9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3" Nov 02 15:36:47.738015 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.737997 4469 generic.go:356] "Generic (PLEG): container finished" podID="225d6774-62eb-46c1-b360-9a6af913b885" containerID="5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1" exitCode=0 Nov 02 15:36:47.738122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.738045 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" event={"ID":"225d6774-62eb-46c1-b360-9a6af913b885","Type":"ContainerDied","Data":"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1"} Nov 02 15:36:47.738122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.738076 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" event={"ID":"225d6774-62eb-46c1-b360-9a6af913b885","Type":"ContainerDied","Data":"150972d7e471d24be010b1d6b65976fdf3178225cd5eea612acd94f9f3658fbf"} Nov 02 15:36:47.738122 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.738046 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h" Nov 02 15:36:47.746399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746377 4469 scope.go:122] "RemoveContainer" containerID="9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3" Nov 02 15:36:47.746649 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746627 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\") pod \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " Nov 02 15:36:47.746723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746672 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\") pod \"225d6774-62eb-46c1-b360-9a6af913b885\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " Nov 02 15:36:47.746723 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:47.746668 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3\": container with ID starting with 9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3 not found: ID does not exist" containerID="9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3" Nov 02 15:36:47.746723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746700 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\" (UniqueName: \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\") pod \"225d6774-62eb-46c1-b360-9a6af913b885\" (UID: \"225d6774-62eb-46c1-b360-9a6af913b885\") " Nov 02 15:36:47.746723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746700 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3"} err="failed to get container status \"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3\": rpc error: code = NotFound desc = could not find container \"9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3\": container with ID starting with 9d317bc16dd742582aff3636a5c0bb8e14be85b18ad267c3223e5426ea691bb3 not found: ID does not exist" Nov 02 15:36:47.746723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746721 4469 scope.go:122] "RemoveContainer" containerID="5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1" Nov 02 15:36:47.747091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.746790 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\" (UniqueName: \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\") pod \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\" (UID: \"9fd1784a-b0ba-490d-83c5-3171a145cbd5\") " Nov 02 15:36:47.747091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.747053 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume" pod "225d6774-62eb-46c1-b360-9a6af913b885" (UID: "225d6774-62eb-46c1-b360-9a6af913b885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:47.747091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.747083 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume" pod "9fd1784a-b0ba-490d-83c5-3171a145cbd5" (UID: "9fd1784a-b0ba-490d-83c5-3171a145cbd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:47.747300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.747282 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd1784a-b0ba-490d-83c5-3171a145cbd5-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:47.747379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.747302 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225d6774-62eb-46c1-b360-9a6af913b885-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:47.749095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.749071 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw" pod "225d6774-62eb-46c1-b360-9a6af913b885" (UID: "225d6774-62eb-46c1-b360-9a6af913b885"). InnerVolumeSpecName "kube-api-access-x6tfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:47.749095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.749079 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt" pod "9fd1784a-b0ba-490d-83c5-3171a145cbd5" (UID: "9fd1784a-b0ba-490d-83c5-3171a145cbd5"). InnerVolumeSpecName "kube-api-access-vdrqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:47.755493 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.755471 4469 scope.go:122] "RemoveContainer" containerID="5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1" Nov 02 15:36:47.755763 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:36:47.755747 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1\": container with ID starting with 5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1 not found: ID does not exist" containerID="5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1" Nov 02 15:36:47.755919 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.755771 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1"} err="failed to get container status \"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1\": rpc error: code = NotFound desc = could not find container \"5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1\": container with ID starting with 5b8aff29f24a26f9aa9325950c49a65fa46145df16e4c477488f1158521fc3d1 not found: ID does not exist" Nov 02 15:36:47.848811 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.848766 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrqt\" (UniqueName: \"kubernetes.io/projected/9fd1784a-b0ba-490d-83c5-3171a145cbd5-kube-api-access-vdrqt\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:47.848977 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:47.848796 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tfw\" (UniqueName: \"kubernetes.io/projected/225d6774-62eb-46c1-b360-9a6af913b885-kube-api-access-x6tfw\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:36:48.162232 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:48.162197 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:36:48.167699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:48.167671 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-db3931-5b4bd8c8c5-g8gwx"] Nov 02 15:36:48.277683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:48.277652 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:36:48.281510 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:48.281481 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db3931-587478dc8f-gbx6h"] Nov 02 15:36:49.238011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:49.237978 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225d6774-62eb-46c1-b360-9a6af913b885" path="/var/lib/kubelet/pods/225d6774-62eb-46c1-b360-9a6af913b885/volumes" Nov 02 15:36:49.238411 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:49.238393 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" path="/var/lib/kubelet/pods/9fd1784a-b0ba-490d-83c5-3171a145cbd5/volumes" Nov 02 15:36:49.639495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:49.639389 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:53.655302 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:53.655260 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:54.296436 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:54.296398 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:54.501252 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:54.501213 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:54.901702 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:54.901665 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:55.114931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:55.114897 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:55.625656 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:55.625619 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:56.836228 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:56.836193 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:36:56.878818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:56.878764 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:59.501483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:36:59.501445 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:02.116275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:02.116236 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:02.334590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:02.334551 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:04.466657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:04.466616 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:04.837383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:04.837301 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:05.044784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:05.044744 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:06.188659 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:37:06.188622 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:37:09.350286 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:09.350249 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:10.738312 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:10.738274 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:14.271048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:14.271015 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:18.638050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:18.638013 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:20.160935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:20.160902 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:20.461908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:20.461880 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:20.678504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:20.678466 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:21.214944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:21.214912 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:24.286550 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:24.286512 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:25.699426 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:25.699387 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:27.901576 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:27.900493 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:28.114326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:28.114275 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:28.175041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:28.174968 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:37:28.175196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:28.175173 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" containerID="cri-o://7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7" gracePeriod=30 Nov 02 15:37:28.328665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:28.328630 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:29.404520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:29.404485 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:30.652756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:30.652721 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:33.454665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.454594 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:37:33.584767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584734 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.584966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584773 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.584966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584826 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.584966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584860 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.584966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584905 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.585220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.584986 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.585220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585017 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.585220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585092 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.585220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585134 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:33.585220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585144 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:33.585480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585221 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\") pod \"b9e42948-0742-419f-8e69-7836d124cdd0\" (UID: \"b9e42948-0742-419f-8e69-7836d124cdd0\") " Nov 02 15:37:33.585480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585253 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:33.585480 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585270 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:33.585646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585521 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:33.585719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585672 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.585719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585691 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e42948-0742-419f-8e69-7836d124cdd0-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.585719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585706 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.585902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585722 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.585902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.585735 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.587437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.587409 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "kube-api-access-r4chw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:33.587549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.587454 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:37:33.587757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.587737 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:33.587847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.587758 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token" pod "b9e42948-0742-419f-8e69-7836d124cdd0" (UID: "b9e42948-0742-419f-8e69-7836d124cdd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:33.687116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.687081 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4chw\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-kube-api-access-r4chw\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.687116 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.687111 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e42948-0742-419f-8e69-7836d124cdd0-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.687313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.687124 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e42948-0742-419f-8e69-7836d124cdd0-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.687313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.687137 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e42948-0742-419f-8e69-7836d124cdd0-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:33.994903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.994865 4469 generic.go:356] "Generic (PLEG): container finished" podID="b9e42948-0742-419f-8e69-7836d124cdd0" containerID="7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7" exitCode=0 Nov 02 15:37:33.995090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.994911 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" Nov 02 15:37:33.995090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.994953 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" event={"ID":"b9e42948-0742-419f-8e69-7836d124cdd0","Type":"ContainerDied","Data":"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7"} Nov 02 15:37:33.995090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.995000 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8" event={"ID":"b9e42948-0742-419f-8e69-7836d124cdd0","Type":"ContainerDied","Data":"0781ffa4ea0969aca9f5af02ea14881da2bb89c06b460db2d2b15467562ef50d"} Nov 02 15:37:33.995090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:33.995020 4469 scope.go:122] "RemoveContainer" containerID="7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7" Nov 02 15:37:34.005177 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:34.005158 4469 scope.go:122] "RemoveContainer" containerID="7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7" Nov 02 15:37:34.005449 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:37:34.005427 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7\": container with ID starting with 7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7 not found: ID does not exist" containerID="7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7" Nov 02 15:37:34.005495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:34.005456 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7"} err="failed to get container status \"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7\": rpc error: code = NotFound desc = could not find container \"7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7\": container with ID starting with 7819df4b2f193da97c5b1850afc985c3af189f9edd85651af702394f7a92a5c7 not found: ID does not exist" Nov 02 15:37:34.162062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:34.162023 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:37:34.210061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:34.210030 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-mm2v8"] Nov 02 15:37:35.240433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:35.240391 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" path="/var/lib/kubelet/pods/b9e42948-0742-419f-8e69-7836d124cdd0/volumes" Nov 02 15:37:35.646343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:35.646255 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:35.851941 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:35.851907 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:37:36.186721 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:37:36.186688 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:37:36.650337 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:36.650297 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:36.856824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:36.856300 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:40.748070 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:40.748028 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:45.585875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:45.585841 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:47.319071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:47.319040 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:50.560823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:50.560726 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:54.820893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.820852 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5"] Nov 02 15:37:54.821308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821295 4469 cpu_manager.go:414] "containerMap: removing container" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821309 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821318 4469 cpu_manager.go:414] "containerMap: removing container" podUID="7d2fba0b-1a94-4eba-a6c5-157939eeb911" containerName="revoke-keys" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821322 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2fba0b-1a94-4eba-a6c5-157939eeb911" containerName="revoke-keys" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821339 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821344 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" Nov 02 15:37:54.821358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821359 4469 cpu_manager.go:414] "containerMap: removing container" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821364 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821376 4469 cpu_manager.go:414] "containerMap: removing container" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821381 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28912ade-3ae0-4421-b891-da992963b763" containerName="istio-proxy" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821449 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="225d6774-62eb-46c1-b360-9a6af913b885" containerName="payload-processing" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821460 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2fba0b-1a94-4eba-a6c5-157939eeb911" containerName="revoke-keys" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821467 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e42948-0742-419f-8e69-7836d124cdd0" containerName="istio-proxy" Nov 02 15:37:54.821544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.821475 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd1784a-b0ba-490d-83c5-3171a145cbd5" containerName="payload-pre-processing" Nov 02 15:37:54.826140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.826108 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.852439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.852411 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.852621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.852478 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58r6j\" (UniqueName: \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.852696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.852664 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.884740 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.884710 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5"] Nov 02 15:37:54.927935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.927899 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:37:54.928083 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.927953 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:37:54.953445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.953410 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.953676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.953484 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.953676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.953522 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58r6j\" (UniqueName: \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.954357 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.954326 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:54.955959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:54.955936 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:55.014662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:55.014629 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58r6j\" (UniqueName: \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\") pod \"maas-api-revoke-sub-ebb4597b-wvsg5\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:55.141668 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:55.141582 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:55.518839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:55.518814 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5"] Nov 02 15:37:55.520450 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:37:55.520419 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283dda69_9e5d_441c_addc_c786a41c12d0.slice/crio-50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0 WatchSource:0}: Error finding container 50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0: Status 404 returned error can't find the container with id 50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0 Nov 02 15:37:55.653310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:55.653266 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:56.109432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:56.109352 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" event={"ID":"283dda69-9e5d-441c-addc-c786a41c12d0","Type":"ContainerStarted","Data":"b5d92c4ab72e3269e3d9d5ee5ba1e87c3c5c2cf01cc6f1581a1f6e6a55cacc70"} Nov 02 15:37:56.109821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:56.109436 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" event={"ID":"283dda69-9e5d-441c-addc-c786a41c12d0","Type":"ContainerStarted","Data":"50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0"} Nov 02 15:37:56.225836 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:56.225766 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" podStartSLOduration=2.225752383 podStartE2EDuration="2.225752383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:56.224288313 +0000 UTC m=+2479.107890854" watchObservedRunningTime="2026-11-02 15:37:56.225752383 +0000 UTC m=+2479.109354918" Nov 02 15:37:57.116078 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:57.116042 4469 generic.go:356] "Generic (PLEG): container finished" podID="283dda69-9e5d-441c-addc-c786a41c12d0" containerID="b5d92c4ab72e3269e3d9d5ee5ba1e87c3c5c2cf01cc6f1581a1f6e6a55cacc70" exitCode=0 Nov 02 15:37:57.116463 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:57.116132 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" event={"ID":"283dda69-9e5d-441c-addc-c786a41c12d0","Type":"ContainerDied","Data":"b5d92c4ab72e3269e3d9d5ee5ba1e87c3c5c2cf01cc6f1581a1f6e6a55cacc70"} Nov 02 15:37:57.218994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:57.218963 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:58.276520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.276492 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:58.386708 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.386618 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\") pod \"283dda69-9e5d-441c-addc-c786a41c12d0\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " Nov 02 15:37:58.386878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.386717 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\") pod \"283dda69-9e5d-441c-addc-c786a41c12d0\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " Nov 02 15:37:58.386878 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.386812 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\" (UniqueName: \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\") pod \"283dda69-9e5d-441c-addc-c786a41c12d0\" (UID: \"283dda69-9e5d-441c-addc-c786a41c12d0\") " Nov 02 15:37:58.387012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.386984 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca" pod "283dda69-9e5d-441c-addc-c786a41c12d0" (UID: "283dda69-9e5d-441c-addc-c786a41c12d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:58.387292 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.387266 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:58.388731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.388703 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca" pod "283dda69-9e5d-441c-addc-c786a41c12d0" (UID: "283dda69-9e5d-441c-addc-c786a41c12d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:58.388940 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.388920 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j" pod "283dda69-9e5d-441c-addc-c786a41c12d0" (UID: "283dda69-9e5d-441c-addc-c786a41c12d0"). InnerVolumeSpecName "kube-api-access-58r6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:58.487982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.487946 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283dda69-9e5d-441c-addc-c786a41c12d0-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:58.487982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:58.487975 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58r6j\" (UniqueName: \"kubernetes.io/projected/283dda69-9e5d-441c-addc-c786a41c12d0-kube-api-access-58r6j\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:37:59.128058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:59.128025 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" Nov 02 15:37:59.128231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:59.128062 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5" event={"ID":"283dda69-9e5d-441c-addc-c786a41c12d0","Type":"ContainerDied","Data":"50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0"} Nov 02 15:37:59.128231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:37:59.128157 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e5a59c790d87c43f716cf465895f66dce728a13972eb84dd32539878dd76a0" Nov 02 15:38:00.572003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:00.571966 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:05.584910 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:05.584865 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:06.187347 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:38:06.187313 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:38:06.851770 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:06.851731 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:07.099111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:07.099081 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:08.455491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:08.455452 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:10.582149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:10.582108 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:11.088385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:11.088326 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:14.515989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:14.515954 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:38:14.583592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:14.583562 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:14.815082 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:14.814996 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:15.472186 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:15.472058 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:17.620867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:17.620791 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:38:17.621333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:17.621094 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" containerID="cri-o://1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7" gracePeriod=30 Nov 02 15:38:20.447868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:20.447826 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:22.873413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.873388 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:38:22.962819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962774 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.962954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962849 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.962954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962887 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.962954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962929 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.962954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962945 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.963187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.962996 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.963187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963126 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.963187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963160 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.963345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963187 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\") pod \"b614d002-216c-4755-acb3-f723b729bb82\" (UID: \"b614d002-216c-4755-acb3-f723b729bb82\") " Nov 02 15:38:22.963345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963240 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:22.963345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963248 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:22.963345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963269 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:22.963562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963448 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:22.963562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963528 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:22.963562 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963553 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:22.963734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963574 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:22.963734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963588 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:22.963734 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.963597 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:22.965094 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.965072 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "kube-api-access-bmx28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:22.965214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.965173 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:22.965490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.965471 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:22.965553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:22.965488 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token" pod "b614d002-216c-4755-acb3-f723b729bb82" (UID: "b614d002-216c-4755-acb3-f723b729bb82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:23.064676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.064647 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b614d002-216c-4755-acb3-f723b729bb82-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:23.064829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.064681 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b614d002-216c-4755-acb3-f723b729bb82-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:23.064829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.064696 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmx28\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-kube-api-access-bmx28\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:23.064829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.064711 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b614d002-216c-4755-acb3-f723b729bb82-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:23.064829 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.064727 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b614d002-216c-4755-acb3-f723b729bb82-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:23.256651 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.256620 4469 generic.go:356] "Generic (PLEG): container finished" podID="b614d002-216c-4755-acb3-f723b729bb82" containerID="1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7" exitCode=0 Nov 02 15:38:23.256757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.256661 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" event={"ID":"b614d002-216c-4755-acb3-f723b729bb82","Type":"ContainerDied","Data":"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7"} Nov 02 15:38:23.256757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.256671 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" Nov 02 15:38:23.256757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.256707 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc" event={"ID":"b614d002-216c-4755-acb3-f723b729bb82","Type":"ContainerDied","Data":"e11cdcaaebba1a8a0bdd65d6d2fd17db94a5c1688546a452411a7dfcf3af1ff8"} Nov 02 15:38:23.256757 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.256731 4469 scope.go:122] "RemoveContainer" containerID="1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7" Nov 02 15:38:23.265636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.265617 4469 scope.go:122] "RemoveContainer" containerID="1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7" Nov 02 15:38:23.265941 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:38:23.265922 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7\": container with ID starting with 1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7 not found: ID does not exist" containerID="1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7" Nov 02 15:38:23.266003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.265952 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7"} err="failed to get container status \"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7\": rpc error: code = NotFound desc = could not find container \"1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7\": container with ID starting with 1006dc29f275c83d0d9e6a1aad90f4cddf7bb64225942a759b11df67b8912cb7 not found: ID does not exist" Nov 02 15:38:23.431581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.431549 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:38:23.480553 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:23.480525 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-9rhdc"] Nov 02 15:38:25.246114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:25.246076 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b614d002-216c-4755-acb3-f723b729bb82" path="/var/lib/kubelet/pods/b614d002-216c-4755-acb3-f723b729bb82/volumes" Nov 02 15:38:25.251618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:25.251587 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:29.296776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.296743 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:38:29.297399 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.297009 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" podUID="88049ad1-b644-4eda-87cd-928cc3570c27" containerName="maas-api" containerID="cri-o://5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d" gracePeriod=30 Nov 02 15:38:29.483634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.483595 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:29.575976 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.575944 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:38:29.653137 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.653104 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") pod \"88049ad1-b644-4eda-87cd-928cc3570c27\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " Nov 02 15:38:29.653326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.653148 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\") pod \"88049ad1-b644-4eda-87cd-928cc3570c27\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " Nov 02 15:38:29.653326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.653271 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\") pod \"88049ad1-b644-4eda-87cd-928cc3570c27\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " Nov 02 15:38:29.653326 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.653321 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\" (UniqueName: \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\") pod \"88049ad1-b644-4eda-87cd-928cc3570c27\" (UID: \"88049ad1-b644-4eda-87cd-928cc3570c27\") " Nov 02 15:38:29.653762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.653735 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca" pod "88049ad1-b644-4eda-87cd-928cc3570c27" (UID: "88049ad1-b644-4eda-87cd-928cc3570c27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:29.655396 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.655366 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs" pod "88049ad1-b644-4eda-87cd-928cc3570c27" (UID: "88049ad1-b644-4eda-87cd-928cc3570c27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:29.655491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.655391 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls" pod "88049ad1-b644-4eda-87cd-928cc3570c27" (UID: "88049ad1-b644-4eda-87cd-928cc3570c27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:29.655491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.655424 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l" pod "88049ad1-b644-4eda-87cd-928cc3570c27" (UID: "88049ad1-b644-4eda-87cd-928cc3570c27"). InnerVolumeSpecName "kube-api-access-kzg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:29.754385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.754350 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzg2l\" (UniqueName: \"kubernetes.io/projected/88049ad1-b644-4eda-87cd-928cc3570c27-kube-api-access-kzg2l\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:29.754385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.754385 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:29.754619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.754402 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88049ad1-b644-4eda-87cd-928cc3570c27-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:29.754619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.754416 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88049ad1-b644-4eda-87cd-928cc3570c27-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:29.803868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.803766 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:38:29.804343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:29.804251 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" containerID="cri-o://4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f" gracePeriod=30 Nov 02 15:38:30.121197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.121111 4469 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:38:30.121383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.121197 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:38:30.302482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.302430 4469 generic.go:356] "Generic (PLEG): container finished" podID="88049ad1-b644-4eda-87cd-928cc3570c27" containerID="5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d" exitCode=0 Nov 02 15:38:30.303477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.302897 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" event={"ID":"88049ad1-b644-4eda-87cd-928cc3570c27","Type":"ContainerDied","Data":"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d"} Nov 02 15:38:30.303477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.303064 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" event={"ID":"88049ad1-b644-4eda-87cd-928cc3570c27","Type":"ContainerDied","Data":"51ad7aa4abf9d82944070f2614aa9c56d4ce53201938544e2986a90377b86d10"} Nov 02 15:38:30.303477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.303230 4469 scope.go:122] "RemoveContainer" containerID="5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d" Nov 02 15:38:30.304270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.303583 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf" Nov 02 15:38:30.337285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.337259 4469 scope.go:122] "RemoveContainer" containerID="5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d" Nov 02 15:38:30.337864 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:38:30.337841 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d\": container with ID starting with 5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d not found: ID does not exist" containerID="5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d" Nov 02 15:38:30.338080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.338052 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d"} err="failed to get container status \"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d\": rpc error: code = NotFound desc = could not find container \"5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d\": container with ID starting with 5dd682ac9b65eee9ffce88e7266f37e461af845362539053278bd9e8c6903a3d not found: ID does not exist" Nov 02 15:38:30.484128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.484097 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:38:30.489706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:30.489680 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c18e79-78d6866876-gpdhf"] Nov 02 15:38:31.047025 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:31.046997 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:31.239392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:31.239348 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88049ad1-b644-4eda-87cd-928cc3570c27" path="/var/lib/kubelet/pods/88049ad1-b644-4eda-87cd-928cc3570c27/volumes" Nov 02 15:38:31.271127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:31.271092 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:31.485677 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:31.485642 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:32.014204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:32.014158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:35.070971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.070944 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:38:35.116178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116092 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116146 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116174 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116218 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116236 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116273 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116314 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116345 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116407 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\") pod \"72b59b9d-f350-482b-8df2-e081452daefc\" (UID: \"72b59b9d-f350-482b-8df2-e081452daefc\") " Nov 02 15:38:35.116845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116545 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:35.116845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116565 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:35.116845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116663 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:35.116845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116730 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:35.117071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116859 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b59b9d-f350-482b-8df2-e081452daefc-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.117071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116999 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.117071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.117015 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.117071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.116813 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:35.118648 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.118614 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:35.118780 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.118710 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:35.118901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.118879 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:35.118943 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.118888 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft" pod "72b59b9d-f350-482b-8df2-e081452daefc" (UID: "72b59b9d-f350-482b-8df2-e081452daefc"). InnerVolumeSpecName "kube-api-access-x4xft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:35.217979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.217942 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xft\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-kube-api-access-x4xft\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.217979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.217973 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.217979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.217986 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.218218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.217998 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b59b9d-f350-482b-8df2-e081452daefc-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.218218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.218009 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b59b9d-f350-482b-8df2-e081452daefc-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.218218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.218021 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b59b9d-f350-482b-8df2-e081452daefc-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:38:35.363911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.363873 4469 generic.go:356] "Generic (PLEG): container finished" podID="72b59b9d-f350-482b-8df2-e081452daefc" containerID="4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f" exitCode=0 Nov 02 15:38:35.364089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.363924 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" Nov 02 15:38:35.364089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.363954 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" event={"ID":"72b59b9d-f350-482b-8df2-e081452daefc","Type":"ContainerDied","Data":"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f"} Nov 02 15:38:35.364089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.364018 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt" event={"ID":"72b59b9d-f350-482b-8df2-e081452daefc","Type":"ContainerDied","Data":"d7ba3702861d16a42c8b9cd022818c332514436f9551c8f6b9686995e29f955b"} Nov 02 15:38:35.364089 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.364048 4469 scope.go:122] "RemoveContainer" containerID="4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f" Nov 02 15:38:35.373137 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.373117 4469 scope.go:122] "RemoveContainer" containerID="4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f" Nov 02 15:38:35.373406 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:38:35.373386 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f\": container with ID starting with 4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f not found: ID does not exist" containerID="4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f" Nov 02 15:38:35.373447 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.373416 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f"} err="failed to get container status \"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f\": rpc error: code = NotFound desc = could not find container \"4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f\": container with ID starting with 4fc148ab53ce91667820cffd14fc95ef0dd749e387f3c53a8d46aa1c2054453f not found: ID does not exist" Nov 02 15:38:35.529535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.529502 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:38:35.531828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:35.531792 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-bj5xt"] Nov 02 15:38:36.186822 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:38:36.186770 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:38:36.655828 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:36.655329 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:37.240778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:37.240739 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b59b9d-f350-482b-8df2-e081452daefc" path="/var/lib/kubelet/pods/72b59b9d-f350-482b-8df2-e081452daefc/volumes" Nov 02 15:38:38.473984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:38.473949 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:38:39.014190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:39.014145 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:39.224681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:39.224644 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:39.441902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:39.441825 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:41.556029 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:41.549449 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:46.379783 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:46.379727 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:47.662591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:47.662548 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:51.264822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:51.261873 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:54.244685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:54.244648 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:54.447201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:54.447169 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:56.088673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:56.088634 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:56.877837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:56.877782 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:57.557949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.557912 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:57.833037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.832963 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr"] Nov 02 15:38:57.833387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833375 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" Nov 02 15:38:57.833438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833388 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" Nov 02 15:38:57.833438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833397 4469 cpu_manager.go:414] "containerMap: removing container" podUID="283dda69-9e5d-441c-addc-c786a41c12d0" containerName="revoke-keys" Nov 02 15:38:57.833438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833402 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283dda69-9e5d-441c-addc-c786a41c12d0" containerName="revoke-keys" Nov 02 15:38:57.833438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833430 4469 cpu_manager.go:414] "containerMap: removing container" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" Nov 02 15:38:57.833438 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833435 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833442 4469 cpu_manager.go:414] "containerMap: removing container" podUID="88049ad1-b644-4eda-87cd-928cc3570c27" containerName="maas-api" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833449 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88049ad1-b644-4eda-87cd-928cc3570c27" containerName="maas-api" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833515 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b59b9d-f350-482b-8df2-e081452daefc" containerName="istio-proxy" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833522 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="88049ad1-b644-4eda-87cd-928cc3570c27" containerName="maas-api" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833532 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b614d002-216c-4755-acb3-f723b729bb82" containerName="istio-proxy" Nov 02 15:38:57.833613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.833537 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="283dda69-9e5d-441c-addc-c786a41c12d0" containerName="revoke-keys" Nov 02 15:38:57.838066 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.838048 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.868813 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.868760 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k88d\" (UniqueName: \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.868951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.868831 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.868951 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.868919 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.896213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.896181 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr"] Nov 02 15:38:57.942528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.942499 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:38:57.942659 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.942539 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:38:57.970371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.970345 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k88d\" (UniqueName: \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.970516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.970388 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.970516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.970415 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.971231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.971205 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:57.972726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:57.972703 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:58.031693 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:58.031662 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k88d\" (UniqueName: \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\") pod \"maas-api-revoke-sub-2c9897c1-45tfr\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:58.150975 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:58.150900 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:38:58.545706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:58.545675 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr"] Nov 02 15:38:58.546919 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:38:58.546888 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f74ab_3edd_4a1f_83a7_ea59d840ff9c.slice/crio-36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8 WatchSource:0}: Error finding container 36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8: Status 404 returned error can't find the container with id 36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8 Nov 02 15:38:59.492408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:59.492377 4469 generic.go:356] "Generic (PLEG): container finished" podID="179f74ab-3edd-4a1f-83a7-ea59d840ff9c" containerID="2f063bfdf51e25461bf69cf8e5d25dbf55f2610ab0d9757e1840f137c44284ea" exitCode=0 Nov 02 15:38:59.492782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:59.492420 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" event={"ID":"179f74ab-3edd-4a1f-83a7-ea59d840ff9c","Type":"ContainerDied","Data":"2f063bfdf51e25461bf69cf8e5d25dbf55f2610ab0d9757e1840f137c44284ea"} Nov 02 15:38:59.492782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:38:59.492460 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" event={"ID":"179f74ab-3edd-4a1f-83a7-ea59d840ff9c","Type":"ContainerStarted","Data":"36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8"} Nov 02 15:39:00.640118 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.640093 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:39:00.698695 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.698667 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\" (UniqueName: \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\") pod \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " Nov 02 15:39:00.698855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.698769 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\") pod \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " Nov 02 15:39:00.698855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.698788 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\") pod \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\" (UID: \"179f74ab-3edd-4a1f-83a7-ea59d840ff9c\") " Nov 02 15:39:00.699159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.699133 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca" pod "179f74ab-3edd-4a1f-83a7-ea59d840ff9c" (UID: "179f74ab-3edd-4a1f-83a7-ea59d840ff9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:00.700705 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.700684 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d" pod "179f74ab-3edd-4a1f-83a7-ea59d840ff9c" (UID: "179f74ab-3edd-4a1f-83a7-ea59d840ff9c"). InnerVolumeSpecName "kube-api-access-6k88d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:00.700811 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.700779 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca" pod "179f74ab-3edd-4a1f-83a7-ea59d840ff9c" (UID: "179f74ab-3edd-4a1f-83a7-ea59d840ff9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:00.800599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.800525 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:00.800599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.800557 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:00.800599 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.800569 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k88d\" (UniqueName: \"kubernetes.io/projected/179f74ab-3edd-4a1f-83a7-ea59d840ff9c-kube-api-access-6k88d\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:00.874566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:00.873297 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:01.504133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:01.504062 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" Nov 02 15:39:01.504283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:01.504056 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr" event={"ID":"179f74ab-3edd-4a1f-83a7-ea59d840ff9c","Type":"ContainerDied","Data":"36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8"} Nov 02 15:39:01.504283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:01.504185 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b9c8b103e396e8cac6562be2f8d8292d850466bd5c85702304bfac4aafe6f8" Nov 02 15:39:05.767187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:05.767149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:06.185124 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:39:06.185050 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:39:09.958027 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:09.957987 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:10.684171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.684129 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:10.894153 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.894120 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv"] Nov 02 15:39:10.894580 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.894567 4469 cpu_manager.go:414] "containerMap: removing container" podUID="179f74ab-3edd-4a1f-83a7-ea59d840ff9c" containerName="revoke-keys" Nov 02 15:39:10.894627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.894581 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179f74ab-3edd-4a1f-83a7-ea59d840ff9c" containerName="revoke-keys" Nov 02 15:39:10.894687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.894678 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="179f74ab-3edd-4a1f-83a7-ea59d840ff9c" containerName="revoke-keys" Nov 02 15:39:10.897773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.897755 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:10.999457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.999426 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:39:10.999457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:10.999434 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:39:11.001748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.001726 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv"] Nov 02 15:39:11.008393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.008364 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjn7m\" (UniqueName: \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.008515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.008396 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.008515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.008497 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.109496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.109472 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjn7m\" (UniqueName: \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.109632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.109503 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.109632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.109553 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.110206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.110189 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.111814 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.111786 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.170619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.170591 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjn7m\" (UniqueName: \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\") pod \"maas-api-revoke-sub-f5aea62b-4qrvv\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.210634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.210600 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:11.523385 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.523314 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:11.584200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:11.584177 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv"] Nov 02 15:39:11.586076 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:39:11.586052 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a7bda7_68cd_44b2_8974_aaf5382c3631.slice/crio-e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c WatchSource:0}: Error finding container e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c: Status 404 returned error can't find the container with id e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c Nov 02 15:39:12.569156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:12.569119 4469 generic.go:356] "Generic (PLEG): container finished" podID="e2a7bda7-68cd-44b2-8974-aaf5382c3631" containerID="bc57018fef2421dc37f39b168f86f9a678413acc54bd2af3037bb048d398cf33" exitCode=0 Nov 02 15:39:12.569542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:12.569171 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" event={"ID":"e2a7bda7-68cd-44b2-8974-aaf5382c3631","Type":"ContainerDied","Data":"bc57018fef2421dc37f39b168f86f9a678413acc54bd2af3037bb048d398cf33"} Nov 02 15:39:12.569542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:12.569219 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" event={"ID":"e2a7bda7-68cd-44b2-8974-aaf5382c3631","Type":"ContainerStarted","Data":"e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c"} Nov 02 15:39:13.697407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.697380 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:13.836837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.836751 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\" (UniqueName: \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\") pod \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " Nov 02 15:39:13.837001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.836854 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\") pod \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " Nov 02 15:39:13.837001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.836915 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\") pod \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\" (UID: \"e2a7bda7-68cd-44b2-8974-aaf5382c3631\") " Nov 02 15:39:13.837309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.837279 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca" pod "e2a7bda7-68cd-44b2-8974-aaf5382c3631" (UID: "e2a7bda7-68cd-44b2-8974-aaf5382c3631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:13.838870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.838849 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca" pod "e2a7bda7-68cd-44b2-8974-aaf5382c3631" (UID: "e2a7bda7-68cd-44b2-8974-aaf5382c3631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:13.838948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.838920 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m" pod "e2a7bda7-68cd-44b2-8974-aaf5382c3631" (UID: "e2a7bda7-68cd-44b2-8974-aaf5382c3631"). InnerVolumeSpecName "kube-api-access-bjn7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:13.938147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.938115 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:13.938147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.938141 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjn7m\" (UniqueName: \"kubernetes.io/projected/e2a7bda7-68cd-44b2-8974-aaf5382c3631-kube-api-access-bjn7m\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:13.938147 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:13.938152 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a7bda7-68cd-44b2-8974-aaf5382c3631-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:14.581273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:14.581239 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" event={"ID":"e2a7bda7-68cd-44b2-8974-aaf5382c3631","Type":"ContainerDied","Data":"e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c"} Nov 02 15:39:14.581490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:14.581292 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e932fa0121fd2c711a711daabaff87c3c00551472b58847a61541aa1757bc45c" Nov 02 15:39:14.581490 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:14.581248 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv" Nov 02 15:39:15.712333 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:15.712295 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:17.069778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:17.069745 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:39:20.482546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:20.482465 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:25.269323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:25.269285 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:30.155324 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:30.155290 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:34.052126 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:34.052086 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc"] Nov 02 15:39:34.101623 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:34.101589 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f05f9fc-rrkmc"] Nov 02 15:39:34.612158 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:34.612116 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:35.050837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.050789 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4"] Nov 02 15:39:35.051294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.051276 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e2a7bda7-68cd-44b2-8974-aaf5382c3631" containerName="revoke-keys" Nov 02 15:39:35.051383 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.051297 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a7bda7-68cd-44b2-8974-aaf5382c3631" containerName="revoke-keys" Nov 02 15:39:35.051443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.051433 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a7bda7-68cd-44b2-8974-aaf5382c3631" containerName="revoke-keys" Nov 02 15:39:35.055577 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.055556 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.089190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.089162 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:35.118891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.118858 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:39:35.119206 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.119069 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:39:35.121732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.121705 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4"] Nov 02 15:39:35.163138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.163094 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.163469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.163445 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.163665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.163636 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2678m\" (UniqueName: \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.238335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.238303 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94193325-ecad-4c9d-949b-4c1e83cda24f" path="/var/lib/kubelet/pods/94193325-ecad-4c9d-949b-4c1e83cda24f/volumes" Nov 02 15:39:35.264296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.264262 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.264429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.264302 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2678m\" (UniqueName: \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.264429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.264385 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.264993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.264975 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.266479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.266459 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.325581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.325524 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2678m\" (UniqueName: \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\") pod \"maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.368515 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.368487 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:35.742073 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:35.742049 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4"] Nov 02 15:39:35.743927 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:39:35.743899 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c323454_a864_4d73_80c4_6278a59fad79.slice/crio-012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399 WatchSource:0}: Error finding container 012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399: Status 404 returned error can't find the container with id 012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399 Nov 02 15:39:36.186946 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:39:36.186907 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:39:36.712131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:36.712103 4469 generic.go:356] "Generic (PLEG): container finished" podID="0c323454-a864-4d73-80c4-6278a59fad79" containerID="40e9136d62aa428af64aa862b7e63bcb9894469391ec8f68730c0f61130cc37b" exitCode=0 Nov 02 15:39:36.712260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:36.712167 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" event={"ID":"0c323454-a864-4d73-80c4-6278a59fad79","Type":"ContainerDied","Data":"40e9136d62aa428af64aa862b7e63bcb9894469391ec8f68730c0f61130cc37b"} Nov 02 15:39:36.712260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:36.712212 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" event={"ID":"0c323454-a864-4d73-80c4-6278a59fad79","Type":"ContainerStarted","Data":"012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399"} Nov 02 15:39:37.868367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.868340 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:37.994707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.994624 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\" (UniqueName: \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\") pod \"0c323454-a864-4d73-80c4-6278a59fad79\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " Nov 02 15:39:37.994707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.994684 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\") pod \"0c323454-a864-4d73-80c4-6278a59fad79\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " Nov 02 15:39:37.994966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.994729 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\") pod \"0c323454-a864-4d73-80c4-6278a59fad79\" (UID: \"0c323454-a864-4d73-80c4-6278a59fad79\") " Nov 02 15:39:37.995080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.995054 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca" pod "0c323454-a864-4d73-80c4-6278a59fad79" (UID: "0c323454-a864-4d73-80c4-6278a59fad79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:37.995279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.995256 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.996986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.996955 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m" pod "0c323454-a864-4d73-80c4-6278a59fad79" (UID: "0c323454-a864-4d73-80c4-6278a59fad79"). InnerVolumeSpecName "kube-api-access-2678m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.997282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:37.997249 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca" pod "0c323454-a864-4d73-80c4-6278a59fad79" (UID: "0c323454-a864-4d73-80c4-6278a59fad79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:38.096634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.096596 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2678m\" (UniqueName: \"kubernetes.io/projected/0c323454-a864-4d73-80c4-6278a59fad79-kube-api-access-2678m\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:38.096634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.096629 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c323454-a864-4d73-80c4-6278a59fad79-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:39:38.301370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.301290 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:39:38.725685 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.725660 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" Nov 02 15:39:38.725900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.725658 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4" event={"ID":"0c323454-a864-4d73-80c4-6278a59fad79","Type":"ContainerDied","Data":"012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399"} Nov 02 15:39:38.725900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:38.725838 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012510f5c13311059b795f8fceca3983c7ff562c0974e2f4b74cee19617cd399" Nov 02 15:39:40.239361 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:40.239322 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:45.052330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:45.052280 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:47.033573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:47.029692 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:47.249835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:47.249781 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:47.988612 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:47.988575 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:49.966495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:49.966452 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:53.049044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:53.049011 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd"] Nov 02 15:39:53.097315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:53.097288 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880f96ce-bz2xd"] Nov 02 15:39:53.238245 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:53.238210 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec83ff38-a3fa-47e2-b64e-ac01a33d869f" path="/var/lib/kubelet/pods/ec83ff38-a3fa-47e2-b64e-ac01a33d869f/volumes" Nov 02 15:39:54.149262 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:54.149194 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:55.734074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:55.734036 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:55.960091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:55.960059 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:56.169355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:56.169270 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4"] Nov 02 15:39:56.218035 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:56.218003 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-39098c-m5xt4"] Nov 02 15:39:56.304856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:56.304825 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:56.694496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:56.694467 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:57.238470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:57.238432 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c323454-a864-4d73-80c4-6278a59fad79" path="/var/lib/kubelet/pods/0c323454-a864-4d73-80c4-6278a59fad79/volumes" Nov 02 15:39:57.969160 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:39:57.969126 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:00.158565 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:00.158527 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:40:01.346461 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:01.346430 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:03.602046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:03.602017 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:03.848763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:03.848728 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:04.093377 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:04.093346 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:06.184870 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:40:06.184833 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:40:06.387541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:06.387503 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:08.139846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:08.139811 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg"] Nov 02 15:40:08.189204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:08.189171 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ceedac8-pd4lg"] Nov 02 15:40:09.238995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:09.238958 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb92fb1-6349-4596-97f7-0aa30b29f562" path="/var/lib/kubelet/pods/aeb92fb1-6349-4596-97f7-0aa30b29f562/volumes" Nov 02 15:40:11.578309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:11.578268 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:16.295877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:16.295843 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:40:16.446483 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:16.446451 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:16.745286 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:16.745252 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:16.954746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:16.954713 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:21.297099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.297041 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" podUID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" containerName="payload-processing" containerID="cri-o://b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183" gracePeriod=25 Nov 02 15:40:21.361413 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.361381 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:21.547464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.547409 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:40:21.668837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.668785 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\") pod \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " Nov 02 15:40:21.669034 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.668888 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\" (UniqueName: \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\") pod \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\" (UID: \"cf8626d5-72ed-47af-b0f8-2b0863811b8e\") " Nov 02 15:40:21.669197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.669156 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume" pod "cf8626d5-72ed-47af-b0f8-2b0863811b8e" (UID: "cf8626d5-72ed-47af-b0f8-2b0863811b8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:21.669405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.669381 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8626d5-72ed-47af-b0f8-2b0863811b8e-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:40:21.671466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.671443 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx" pod "cf8626d5-72ed-47af-b0f8-2b0863811b8e" (UID: "cf8626d5-72ed-47af-b0f8-2b0863811b8e"). InnerVolumeSpecName "kube-api-access-bjbpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:21.770778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.770745 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbpx\" (UniqueName: \"kubernetes.io/projected/cf8626d5-72ed-47af-b0f8-2b0863811b8e-kube-api-access-bjbpx\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:40:21.799971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.799890 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:21.956471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.956435 4469 generic.go:356] "Generic (PLEG): container finished" podID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" containerID="b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183" exitCode=0 Nov 02 15:40:21.956655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.956482 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" Nov 02 15:40:21.956655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.956525 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" event={"ID":"cf8626d5-72ed-47af-b0f8-2b0863811b8e","Type":"ContainerDied","Data":"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183"} Nov 02 15:40:21.956655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.956585 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5" event={"ID":"cf8626d5-72ed-47af-b0f8-2b0863811b8e","Type":"ContainerDied","Data":"2127ed4e3d6fe7400f03c14930b7c5d599733cc3b9c3126e0f0800ed2b530e37"} Nov 02 15:40:21.956655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.956617 4469 scope.go:122] "RemoveContainer" containerID="b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183" Nov 02 15:40:21.966379 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.966362 4469 scope.go:122] "RemoveContainer" containerID="b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183" Nov 02 15:40:21.966667 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:40:21.966647 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183\": container with ID starting with b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183 not found: ID does not exist" containerID="b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183" Nov 02 15:40:21.966723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:21.966674 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183"} err="failed to get container status \"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183\": rpc error: code = NotFound desc = could not find container \"b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183\": container with ID starting with b9066652571f9222b029640168c9c602ec0189605478bec5bb528da294d23183 not found: ID does not exist" Nov 02 15:40:22.123675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:22.123595 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:40:22.127501 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:22.127479 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4e4683-7784dc776c-n6ch5"] Nov 02 15:40:23.237774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:23.237744 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" path="/var/lib/kubelet/pods/cf8626d5-72ed-47af-b0f8-2b0863811b8e/volumes" Nov 02 15:40:25.595099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:25.595058 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:27.128948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:27.128915 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:27.429437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:27.429329 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:27.632664 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:27.632622 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:28.067067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:28.067026 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:30.699747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:30.699707 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:31.375277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:31.375171 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:40:32.443592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:32.443553 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:34.700891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:34.700488 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:34.908731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:34.908700 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:35.166183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:35.166145 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:36.187584 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:40:36.187538 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:40:37.626605 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:37.626569 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:38.650341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:38.650309 4469 scope.go:122] "RemoveContainer" containerID="6cbe997421e9ec7c5b6a3bb3eb072ba7d06e884d259d542d7e8bbceb949f4d60" Nov 02 15:40:42.776329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:42.776269 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:47.098277 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:47.098219 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:47.344387 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:47.344358 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:40:47.552746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:47.552712 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:47.761889 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:47.761853 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:48.340197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:48.340158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:48.886863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:48.886795 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:51.565390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:51.565352 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:51.840467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:51.840384 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:52.345106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.345061 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" podUID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" containerName="payload-processing" containerID="cri-o://bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93" gracePeriod=25 Nov 02 15:40:52.500381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.500223 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:52.639386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.639364 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:40:52.698735 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.698709 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\" (UniqueName: \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\") pod \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " Nov 02 15:40:52.698925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.698749 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\") pod \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\" (UID: \"fad04e26-7046-41d9-8f11-b9e0a6d0cc01\") " Nov 02 15:40:52.699215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.699190 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume" pod "fad04e26-7046-41d9-8f11-b9e0a6d0cc01" (UID: "fad04e26-7046-41d9-8f11-b9e0a6d0cc01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:52.700857 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.700839 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86" pod "fad04e26-7046-41d9-8f11-b9e0a6d0cc01" (UID: "fad04e26-7046-41d9-8f11-b9e0a6d0cc01"). InnerVolumeSpecName "kube-api-access-ldv86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:52.743203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.743172 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:40:52.799713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.799681 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldv86\" (UniqueName: \"kubernetes.io/projected/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-kube-api-access-ldv86\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:40:52.799713 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:52.799714 4469 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad04e26-7046-41d9-8f11-b9e0a6d0cc01-plugins-config-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:40:53.119371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.119333 4469 generic.go:356] "Generic (PLEG): container finished" podID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" containerID="bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93" exitCode=0 Nov 02 15:40:53.119539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.119385 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" Nov 02 15:40:53.119539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.119400 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" event={"ID":"fad04e26-7046-41d9-8f11-b9e0a6d0cc01","Type":"ContainerDied","Data":"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93"} Nov 02 15:40:53.119539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.119450 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d" event={"ID":"fad04e26-7046-41d9-8f11-b9e0a6d0cc01","Type":"ContainerDied","Data":"f9a3c8ee4687af0354d5d4489fee68db7029162943d3fdbeb5254267c0b0b745"} Nov 02 15:40:53.119539 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.119482 4469 scope.go:122] "RemoveContainer" containerID="bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93" Nov 02 15:40:53.129535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.129515 4469 scope.go:122] "RemoveContainer" containerID="bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93" Nov 02 15:40:53.129840 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:40:53.129792 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93\": container with ID starting with bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93 not found: ID does not exist" containerID="bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93" Nov 02 15:40:53.129918 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.129850 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93"} err="failed to get container status \"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93\": rpc error: code = NotFound desc = could not find container \"bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93\": container with ID starting with bbc8da6a02e2322b9222795ee2d98af52aa1080cc5685d7e49629bea15dd9a93 not found: ID does not exist" Nov 02 15:40:53.285028 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.284998 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:40:53.288894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.288854 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-895ac8-69555fcc47-6n97d"] Nov 02 15:40:53.898441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:53.898402 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:55.237584 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:55.237543 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" path="/var/lib/kubelet/pods/fad04e26-7046-41d9-8f11-b9e0a6d0cc01/volumes" Nov 02 15:40:58.266621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:58.266550 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:58.909962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:58.908323 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:59.230935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:59.230889 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:59.440266 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:40:59.440227 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:03.863946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:03.863910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:06.185330 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:41:06.185290 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:41:08.660745 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:08.660706 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:09.541959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:09.541928 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:41:09.542205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:09.542179 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" containerID="cri-o://acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd" gracePeriod=30 Nov 02 15:41:09.606964 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:09.606936 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:11.101581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:11.101543 4469 patch_prober.go:29] interesting pod/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:41:11.102053 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:11.101617 4469 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:41:13.892307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:13.892265 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:14.798255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.798226 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:41:14.923124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923054 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923096 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923118 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923142 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923190 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923223 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923253 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923313 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923354 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923430 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\") pod \"2cbc2baa-54da-4420-967e-88fab3fe2981\" (UID: \"2cbc2baa-54da-4420-967e-88fab3fe2981\") " Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923479 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923492 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923504 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:14.923739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923685 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:14.924408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923854 4469 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-credential-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:14.924408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923887 4469 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cbc2baa-54da-4420-967e-88fab3fe2981-istiod-ca-cert\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:14.924408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923902 4469 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-socket\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:14.924408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923916 4469 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-data\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:14.924408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.923930 4469 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-workload-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:14.925341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.925313 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:41:14.925537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.925515 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:14.925591 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.925534 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "kube-api-access-5c52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:14.925631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:14.925610 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token" pod "2cbc2baa-54da-4420-967e-88fab3fe2981" (UID: "2cbc2baa-54da-4420-967e-88fab3fe2981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:15.025371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.025335 4469 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-envoy\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:15.025371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.025366 4469 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-podinfo\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:15.025371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.025377 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c52t\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-kube-api-access-5c52t\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:15.025611 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.025388 4469 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cbc2baa-54da-4420-967e-88fab3fe2981-istio-token\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:15.247929 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.247897 4469 generic.go:356] "Generic (PLEG): container finished" podID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerID="acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd" exitCode=0 Nov 02 15:41:15.248055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.247947 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" Nov 02 15:41:15.248055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.247974 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerDied","Data":"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd"} Nov 02 15:41:15.248055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.248024 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g" event={"ID":"2cbc2baa-54da-4420-967e-88fab3fe2981","Type":"ContainerDied","Data":"fb1b0090be2b8a8b8eea6e97c9ecb7621cae82fd95405b0451ae250934b38246"} Nov 02 15:41:15.248055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.248044 4469 scope.go:122] "RemoveContainer" containerID="acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd" Nov 02 15:41:15.256827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.256791 4469 scope.go:122] "RemoveContainer" containerID="0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc" Nov 02 15:41:15.264530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.264505 4469 scope.go:122] "RemoveContainer" containerID="acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd" Nov 02 15:41:15.264824 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:41:15.264785 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd\": container with ID starting with acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd not found: ID does not exist" containerID="acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd" Nov 02 15:41:15.264907 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.264827 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd"} err="failed to get container status \"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd\": rpc error: code = NotFound desc = could not find container \"acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd\": container with ID starting with acb7c687c70965234e8f906075d392226eb3da0d5df32cef8338841af2a34afd not found: ID does not exist" Nov 02 15:41:15.264907 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.264845 4469 scope.go:122] "RemoveContainer" containerID="0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc" Nov 02 15:41:15.265093 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:41:15.265075 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc\": container with ID starting with 0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc not found: ID does not exist" containerID="0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc" Nov 02 15:41:15.265133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.265101 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc"} err="failed to get container status \"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc\": rpc error: code = NotFound desc = could not find container \"0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc\": container with ID starting with 0002d9bb767633588a8462035faf5de66c802b32b90d7544e9ec3707b00285cc not found: ID does not exist" Nov 02 15:41:15.417955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.417922 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:41:15.467528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:15.467493 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-895ac8-openshift-default-756985bc89-9bq5g"] Nov 02 15:41:17.237386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:17.237344 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" path="/var/lib/kubelet/pods/2cbc2baa-54da-4420-967e-88fab3fe2981/volumes" Nov 02 15:41:18.761665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:18.761632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:22.134446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:22.134416 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:22.358331 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:22.358294 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:22.572796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:22.572758 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:22.948897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:22.948813 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:24.767297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:24.767243 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:26.640347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:26.640307 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:26.991282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:26.991244 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:27.614771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:27.614734 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:29.966731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:29.966683 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:30.743993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:30.743955 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:41:34.351832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:34.351741 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:34.577854 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:34.577822 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:34.802093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:34.802055 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:35.028858 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:35.028817 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:35.236127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:35.236099 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:36.187093 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:41:36.187051 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:41:38.681282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:38.681252 4469 scope.go:122] "RemoveContainer" containerID="029350defde85bcbfbfe304cbc6470548b76f247bb83184c8896a20a373ec79b" Nov 02 15:41:38.691425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:38.691403 4469 scope.go:122] "RemoveContainer" containerID="606e1b4c5f6cbade88207f9c4bf75c3e8c3ff84252f6e5e7e86cfc109b69d1fa" Nov 02 15:41:40.108090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:40.108056 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:41:40.263296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:40.263199 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:42.293304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:42.293267 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:43.055138 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:43.055099 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9"] Nov 02 15:41:43.104494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:43.104456 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9623aa-6vfg9"] Nov 02 15:41:43.238005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:43.237970 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2fba0b-1a94-4eba-a6c5-157939eeb911" path="/var/lib/kubelet/pods/7d2fba0b-1a94-4eba-a6c5-157939eeb911/volumes" Nov 02 15:41:45.139382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:45.139349 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:47.657730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:47.657681 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:47.865446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:47.865410 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:48.699190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:48.699158 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:55.127868 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.127829 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" containerID="cri-o://e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1" gracePeriod=45 Nov 02 15:41:55.375779 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.375754 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:41:55.459501 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.459474 4469 generic.go:356] "Generic (PLEG): container finished" podID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerID="e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1" exitCode=0 Nov 02 15:41:55.459657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.459516 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" Nov 02 15:41:55.459657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.459519 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" event={"ID":"6073da20-cbda-46fa-88c6-42deedeab7b0","Type":"ContainerDied","Data":"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1"} Nov 02 15:41:55.459657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.459555 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9" event={"ID":"6073da20-cbda-46fa-88c6-42deedeab7b0","Type":"ContainerDied","Data":"30ad3487fce3f52ceac993dc5f3aea8640f485ae8786dab00c08ee6749f2a9bc"} Nov 02 15:41:55.459657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.459576 4469 scope.go:122] "RemoveContainer" containerID="e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1" Nov 02 15:41:55.467996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.467980 4469 scope.go:122] "RemoveContainer" containerID="e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1" Nov 02 15:41:55.468260 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:41:55.468240 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1\": container with ID starting with e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1 not found: ID does not exist" containerID="e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1" Nov 02 15:41:55.468328 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.468269 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1"} err="failed to get container status \"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1\": rpc error: code = NotFound desc = could not find container \"e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1\": container with ID starting with e8d792fcf3b57d6dec0a9a378628d687de775f8a4933d179924b017b0e3c62f1 not found: ID does not exist" Nov 02 15:41:55.556289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.556258 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\" (UniqueName: \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.556939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.556307 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.556939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.556350 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.556939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.556390 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.556939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.556422 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.557221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557060 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\") pod \"6073da20-cbda-46fa-88c6-42deedeab7b0\" (UID: \"6073da20-cbda-46fa-88c6-42deedeab7b0\") " Nov 02 15:41:55.557221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557110 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:55.557335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557292 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:55.557432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557414 4469 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-model-cache\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.557492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557436 4469 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-home\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.557637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.557615 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:55.559897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.559871 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:55.559988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.559952 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:55.560271 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.560253 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x" pod "6073da20-cbda-46fa-88c6-42deedeab7b0" (UID: "6073da20-cbda-46fa-88c6-42deedeab7b0"). InnerVolumeSpecName "kube-api-access-7lh6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:55.659111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.659073 4469 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-dshm\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.659111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.659112 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lh6x\" (UniqueName: \"kubernetes.io/projected/6073da20-cbda-46fa-88c6-42deedeab7b0-kube-api-access-7lh6x\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.659308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.659127 4469 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6073da20-cbda-46fa-88c6-42deedeab7b0-tls-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.659308 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.659143 4469 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6073da20-cbda-46fa-88c6-42deedeab7b0-tmp-dir\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:41:55.885150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.885109 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:41:55.889795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:55.889767 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-w1-019efe-kserve-65df55b556-5lzm9"] Nov 02 15:41:57.237615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:57.237585 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" path="/var/lib/kubelet/pods/6073da20-cbda-46fa-88c6-42deedeab7b0/volumes" Nov 02 15:41:58.316282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:58.316248 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:58.671338 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:58.671257 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:59.634863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:41:59.634824 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:00.155547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:00.155514 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:00.461631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:00.461598 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:00.902111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:00.902025 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:04.390615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:04.390584 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:42:04.708968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:04.708891 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:42:04.709128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:04.709106 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" podUID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" containerName="maas-api" containerID="cri-o://723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a" gracePeriod=30 Nov 02 15:42:04.967699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:04.967668 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:04.980616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:04.980596 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:42:05.155526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.155497 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") pod \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " Nov 02 15:42:05.155719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.155565 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\") pod \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " Nov 02 15:42:05.155719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.155585 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\" (UniqueName: \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\") pod \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " Nov 02 15:42:05.155719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.155627 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\") pod \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\" (UID: \"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5\") " Nov 02 15:42:05.156091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.156065 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca" pod "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" (UID: "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:05.157771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.157743 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh" pod "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" (UID: "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5"). InnerVolumeSpecName "kube-api-access-przwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:05.157771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.157754 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls" pod "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" (UID: "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:05.157945 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.157775 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs" pod "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" (UID: "17eced10-4d6a-4f91-ae37-ebcfec5e8ac5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:05.257796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.257707 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:05.257796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.257752 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:05.257796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.257767 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przwh\" (UniqueName: \"kubernetes.io/projected/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-kube-api-access-przwh\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:05.258088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.257827 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:05.511877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.511774 4469 generic.go:356] "Generic (PLEG): container finished" podID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" containerID="723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a" exitCode=0 Nov 02 15:42:05.511877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.511846 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" Nov 02 15:42:05.511877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.511846 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" event={"ID":"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5","Type":"ContainerDied","Data":"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a"} Nov 02 15:42:05.512441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.511927 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2" event={"ID":"17eced10-4d6a-4f91-ae37-ebcfec5e8ac5","Type":"ContainerDied","Data":"e738b7bbdfc7ec24bfeb6400780a2a5b27f0724b0d4c641382a523f985f46256"} Nov 02 15:42:05.512441 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.511951 4469 scope.go:122] "RemoveContainer" containerID="723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a" Nov 02 15:42:05.520970 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.520953 4469 scope.go:122] "RemoveContainer" containerID="723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a" Nov 02 15:42:05.521232 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:42:05.521213 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a\": container with ID starting with 723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a not found: ID does not exist" containerID="723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a" Nov 02 15:42:05.521301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.521239 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a"} err="failed to get container status \"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a\": rpc error: code = NotFound desc = could not find container \"723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a\": container with ID starting with 723daff45d91e84bba57939bd9912447ec741b791323f7c0ec9d9534a195271a not found: ID does not exist" Nov 02 15:42:05.677119 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.677082 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:42:05.681039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:05.681014 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-019efe-66dc6f57dd-97lx2"] Nov 02 15:42:06.184971 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:42:06.184938 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:42:06.470307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:06.470273 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:06.994518 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:06.994488 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:07.240746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:07.240703 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" path="/var/lib/kubelet/pods/17eced10-4d6a-4f91-ae37-ebcfec5e8ac5/volumes" Nov 02 15:42:07.379602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:07.379523 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:08.013149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:08.013109 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:08.245729 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:08.245691 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:13.849732 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:13.849693 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:14.144866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:14.144760 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:14.379404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:14.379368 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:22.486115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:22.486035 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:22.783236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:22.783148 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:42:36.186222 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:42:36.186187 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:42:49.062439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:49.062392 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:53.370126 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370091 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m"] Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370466 4469 cpu_manager.go:414] "containerMap: removing container" podUID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" containerName="payload-processing" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370477 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" containerName="payload-processing" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370491 4469 cpu_manager.go:414] "containerMap: removing container" podUID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" containerName="payload-processing" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370496 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" containerName="payload-processing" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370509 4469 cpu_manager.go:414] "containerMap: removing container" podUID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" containerName="maas-api" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370514 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" containerName="maas-api" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370524 4469 cpu_manager.go:414] "containerMap: removing container" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" Nov 02 15:42:53.370530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370529 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370541 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370546 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370556 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0c323454-a864-4d73-80c4-6278a59fad79" containerName="revoke-keys" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370561 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c323454-a864-4d73-80c4-6278a59fad79" containerName="revoke-keys" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370568 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370572 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370635 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad04e26-7046-41d9-8f11-b9e0a6d0cc01" containerName="payload-processing" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370643 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="17eced10-4d6a-4f91-ae37-ebcfec5e8ac5" containerName="maas-api" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370649 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="6073da20-cbda-46fa-88c6-42deedeab7b0" containerName="main" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370657 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370662 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c323454-a864-4d73-80c4-6278a59fad79" containerName="revoke-keys" Nov 02 15:42:53.370826 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.370670 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8626d5-72ed-47af-b0f8-2b0863811b8e" containerName="payload-processing" Nov 02 15:42:53.374806 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.374782 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.428416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.428382 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:42:53.468711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.468683 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.468865 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.468786 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.468982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.468964 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpp8\" (UniqueName: \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.477658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.477636 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:42:53.477658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.477649 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:42:53.477838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.477737 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:42:53.479585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.479565 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m"] Nov 02 15:42:53.570224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.570192 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.570428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.570231 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.570428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.570307 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpp8\" (UniqueName: \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.570959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.570934 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.572645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.572623 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.631081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.631022 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpp8\" (UniqueName: \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\") pod \"maas-api-revoke-sub-85237e45-rk77m\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:53.687310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:53.687285 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:54.061905 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:54.061713 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m"] Nov 02 15:42:54.064649 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:42:54.064619 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0fa9ad_3702_441d_a727_b15aa1e7412d.slice/crio-c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876 WatchSource:0}: Error finding container c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876: Status 404 returned error can't find the container with id c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876 Nov 02 15:42:54.765794 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:54.765760 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" event={"ID":"db0fa9ad-3702-441d-a727-b15aa1e7412d","Type":"ContainerStarted","Data":"7de8566fec9bf7503a294cf3b8a2b54f33d847435699853964281abbbf486180"} Nov 02 15:42:54.766208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:54.765833 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" event={"ID":"db0fa9ad-3702-441d-a727-b15aa1e7412d","Type":"ContainerStarted","Data":"c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876"} Nov 02 15:42:54.883509 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:54.883462 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" podStartSLOduration=1.883448221 podStartE2EDuration="1.883448221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:54.881820054 +0000 UTC m=+2777.765422574" watchObservedRunningTime="2026-11-02 15:42:54.883448221 +0000 UTC m=+2777.767050739" Nov 02 15:42:55.772491 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:55.772453 4469 generic.go:356] "Generic (PLEG): container finished" podID="db0fa9ad-3702-441d-a727-b15aa1e7412d" containerID="7de8566fec9bf7503a294cf3b8a2b54f33d847435699853964281abbbf486180" exitCode=0 Nov 02 15:42:55.772959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:55.772539 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" event={"ID":"db0fa9ad-3702-441d-a727-b15aa1e7412d","Type":"ContainerDied","Data":"7de8566fec9bf7503a294cf3b8a2b54f33d847435699853964281abbbf486180"} Nov 02 15:42:56.911111 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:56.911087 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:57.003092 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.003062 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\" (UniqueName: \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\") pod \"db0fa9ad-3702-441d-a727-b15aa1e7412d\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " Nov 02 15:42:57.003270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.003248 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\") pod \"db0fa9ad-3702-441d-a727-b15aa1e7412d\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " Nov 02 15:42:57.003329 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.003312 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\") pod \"db0fa9ad-3702-441d-a727-b15aa1e7412d\" (UID: \"db0fa9ad-3702-441d-a727-b15aa1e7412d\") " Nov 02 15:42:57.003622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.003599 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca" pod "db0fa9ad-3702-441d-a727-b15aa1e7412d" (UID: "db0fa9ad-3702-441d-a727-b15aa1e7412d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:57.005305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.005276 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca" pod "db0fa9ad-3702-441d-a727-b15aa1e7412d" (UID: "db0fa9ad-3702-441d-a727-b15aa1e7412d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:57.005443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.005418 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8" pod "db0fa9ad-3702-441d-a727-b15aa1e7412d" (UID: "db0fa9ad-3702-441d-a727-b15aa1e7412d"). InnerVolumeSpecName "kube-api-access-twpp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:57.104790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.104710 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:57.104790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.104742 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0fa9ad-3702-441d-a727-b15aa1e7412d-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:57.104790 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.104752 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpp8\" (UniqueName: \"kubernetes.io/projected/db0fa9ad-3702-441d-a727-b15aa1e7412d-kube-api-access-twpp8\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:42:57.783471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.783432 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" Nov 02 15:42:57.783471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.783460 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m" event={"ID":"db0fa9ad-3702-441d-a727-b15aa1e7412d","Type":"ContainerDied","Data":"c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876"} Nov 02 15:42:57.783704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:57.783540 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59b93c188f01ea7f9bf08cb4723ab821ea0f08c6e21c4d4faaafea79a7f1876" Nov 02 15:42:58.992955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:58.992913 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:59.056143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:59.056101 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5"] Nov 02 15:42:59.105009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:59.104971 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb4597b-wvsg5"] Nov 02 15:42:59.238148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:42:59.238111 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283dda69-9e5d-441c-addc-c786a41c12d0" path="/var/lib/kubelet/pods/283dda69-9e5d-441c-addc-c786a41c12d0/volumes" Nov 02 15:43:06.056287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:06.056254 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:06.185654 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:43:06.185622 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:43:08.272221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:08.272180 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:09.773421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:09.773389 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:09.977038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:09.977005 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:22.863405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:22.863368 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:32.988024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:32.987989 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:33.241348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:33.241268 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:36.184967 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:43:36.184933 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:43:38.739288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:38.739255 4469 scope.go:122] "RemoveContainer" containerID="12252660dbb2f4dc2dc08994fef50e46f12a3b1c35189628926f77bc5e4a89e0" Nov 02 15:43:49.417367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:49.417329 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:51.659889 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:51.659851 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:43:59.354276 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:43:59.354242 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:00.017721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.017683 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2"] Nov 02 15:44:00.018110 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.018096 4469 cpu_manager.go:414] "containerMap: removing container" podUID="db0fa9ad-3702-441d-a727-b15aa1e7412d" containerName="revoke-keys" Nov 02 15:44:00.018157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.018111 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0fa9ad-3702-441d-a727-b15aa1e7412d" containerName="revoke-keys" Nov 02 15:44:00.018197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.018187 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0fa9ad-3702-441d-a727-b15aa1e7412d" containerName="revoke-keys" Nov 02 15:44:00.018239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.018201 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbc2baa-54da-4420-967e-88fab3fe2981" containerName="istio-proxy" Nov 02 15:44:00.022789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.022772 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.060723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.060693 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.060914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.060758 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9wg\" (UniqueName: \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.060997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.060926 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.124671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.124633 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:44:00.124671 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.124635 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:44:00.124926 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.124646 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:44:00.125127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.125111 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:44:00.126744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.126719 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2"] Nov 02 15:44:00.162354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.162331 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9wg\" (UniqueName: \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.162497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.162382 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.162497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.162433 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.163215 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.163193 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.164893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.164861 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.230655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.230631 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9wg\" (UniqueName: \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\") pod \"maas-api-revoke-sub-c1e271a9-krzz2\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.335330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.335261 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:00.712009 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:00.711978 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2"] Nov 02 15:44:00.714181 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:44:00.714157 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode184bd0c_d96b_49a2_b924_985bd9980ceb.slice/crio-c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66 WatchSource:0}: Error finding container c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66: Status 404 returned error can't find the container with id c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66 Nov 02 15:44:01.048972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.048903 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr"] Nov 02 15:44:01.056755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.056733 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9897c1-45tfr"] Nov 02 15:44:01.104691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.104660 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" event={"ID":"e184bd0c-d96b-49a2-b924-985bd9980ceb","Type":"ContainerStarted","Data":"fd4b2d11b9d6e4a7ec14b38a2b68d5726e01f69375429c251df053de516854d7"} Nov 02 15:44:01.104850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.104712 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" event={"ID":"e184bd0c-d96b-49a2-b924-985bd9980ceb","Type":"ContainerStarted","Data":"c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66"} Nov 02 15:44:01.221211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.221165 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" podStartSLOduration=2.221150674 podStartE2EDuration="2.221150674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:01.220420989 +0000 UTC m=+2844.104023509" watchObservedRunningTime="2026-11-02 15:44:01.221150674 +0000 UTC m=+2844.104753193" Nov 02 15:44:01.237555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:01.237523 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179f74ab-3edd-4a1f-83a7-ea59d840ff9c" path="/var/lib/kubelet/pods/179f74ab-3edd-4a1f-83a7-ea59d840ff9c/volumes" Nov 02 15:44:02.111450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:02.111422 4469 generic.go:356] "Generic (PLEG): container finished" podID="e184bd0c-d96b-49a2-b924-985bd9980ceb" containerID="fd4b2d11b9d6e4a7ec14b38a2b68d5726e01f69375429c251df053de516854d7" exitCode=0 Nov 02 15:44:02.111925 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:02.111509 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" event={"ID":"e184bd0c-d96b-49a2-b924-985bd9980ceb","Type":"ContainerDied","Data":"fd4b2d11b9d6e4a7ec14b38a2b68d5726e01f69375429c251df053de516854d7"} Nov 02 15:44:03.250673 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.250647 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:03.293642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.293616 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\") pod \"e184bd0c-d96b-49a2-b924-985bd9980ceb\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " Nov 02 15:44:03.293752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.293656 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\" (UniqueName: \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\") pod \"e184bd0c-d96b-49a2-b924-985bd9980ceb\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " Nov 02 15:44:03.293752 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.293677 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\") pod \"e184bd0c-d96b-49a2-b924-985bd9980ceb\" (UID: \"e184bd0c-d96b-49a2-b924-985bd9980ceb\") " Nov 02 15:44:03.294062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.294039 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca" pod "e184bd0c-d96b-49a2-b924-985bd9980ceb" (UID: "e184bd0c-d96b-49a2-b924-985bd9980ceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:03.295579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.295560 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca" pod "e184bd0c-d96b-49a2-b924-985bd9980ceb" (UID: "e184bd0c-d96b-49a2-b924-985bd9980ceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:03.295711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.295692 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg" pod "e184bd0c-d96b-49a2-b924-985bd9980ceb" (UID: "e184bd0c-d96b-49a2-b924-985bd9980ceb"). InnerVolumeSpecName "kube-api-access-ls9wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:03.396233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.395742 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:03.396233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.395823 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9wg\" (UniqueName: \"kubernetes.io/projected/e184bd0c-d96b-49a2-b924-985bd9980ceb-kube-api-access-ls9wg\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:03.396233 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:03.395846 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e184bd0c-d96b-49a2-b924-985bd9980ceb-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:04.122239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:04.122209 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" Nov 02 15:44:04.122239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:04.122229 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2" event={"ID":"e184bd0c-d96b-49a2-b924-985bd9980ceb","Type":"ContainerDied","Data":"c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66"} Nov 02 15:44:04.122470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:04.122296 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5eb47c24891858731ed79e23fbecd2bcb759f35f409f38003d8998f81cd8a66" Nov 02 15:44:06.184319 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:44:06.184281 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:44:13.170156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:13.170121 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:44:13.279439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:13.279394 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:13.579467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:13.577563 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:14.051250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:14.051216 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv"] Nov 02 15:44:14.100833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:14.100789 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5aea62b-4qrvv"] Nov 02 15:44:15.237336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:15.237304 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a7bda7-68cd-44b2-8974-aaf5382c3631" path="/var/lib/kubelet/pods/e2a7bda7-68cd-44b2-8974-aaf5382c3631/volumes" Nov 02 15:44:23.200715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:23.200677 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:27.286666 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:27.286631 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:44:35.081076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.081040 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb"] Nov 02 15:44:35.081587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.081480 4469 cpu_manager.go:414] "containerMap: removing container" podUID="e184bd0c-d96b-49a2-b924-985bd9980ceb" containerName="revoke-keys" Nov 02 15:44:35.081587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.081490 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e184bd0c-d96b-49a2-b924-985bd9980ceb" containerName="revoke-keys" Nov 02 15:44:35.081587 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.081550 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="e184bd0c-d96b-49a2-b924-985bd9980ceb" containerName="revoke-keys" Nov 02 15:44:35.084924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.084907 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.136779 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.136748 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.136953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.136787 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.136953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.136916 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfb9\" (UniqueName: \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.188003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.187967 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:44:35.188201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.188012 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:44:35.188201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.188024 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:44:35.188201 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.188028 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:44:35.190039 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.190013 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb"] Nov 02 15:44:35.237602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.237564 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.237602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.237602 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.237867 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.237637 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfb9\" (UniqueName: \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.238442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.238417 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.240062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.240042 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.298953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.298924 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfb9\" (UniqueName: \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\") pod \"maas-api-revoke-sub-2e1f5b5f-tfpwb\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.397323 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.397237 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:35.781050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:35.779705 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb"] Nov 02 15:44:35.786423 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:44:35.786394 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd77af1_9903_4d33_89e5_b7ad9590ff61.slice/crio-9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1 WatchSource:0}: Error finding container 9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1: Status 404 returned error can't find the container with id 9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1 Nov 02 15:44:36.186144 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:44:36.186102 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:44:36.299097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:36.299016 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" event={"ID":"1cd77af1-9903-4d33-89e5-b7ad9590ff61","Type":"ContainerStarted","Data":"d8f51315c093a608dfd75f251816bee71d4dce31f9054862b2937c5ba61532f3"} Nov 02 15:44:36.299097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:36.299090 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" event={"ID":"1cd77af1-9903-4d33-89e5-b7ad9590ff61","Type":"ContainerStarted","Data":"9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1"} Nov 02 15:44:37.305052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:37.305022 4469 generic.go:356] "Generic (PLEG): container finished" podID="1cd77af1-9903-4d33-89e5-b7ad9590ff61" containerID="d8f51315c093a608dfd75f251816bee71d4dce31f9054862b2937c5ba61532f3" exitCode=0 Nov 02 15:44:37.305432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:37.305103 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" event={"ID":"1cd77af1-9903-4d33-89e5-b7ad9590ff61","Type":"ContainerDied","Data":"d8f51315c093a608dfd75f251816bee71d4dce31f9054862b2937c5ba61532f3"} Nov 02 15:44:37.929440 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:37.929407 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:38.445011 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.444989 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:38.469358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.469333 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\") pod \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " Nov 02 15:44:38.469492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.469436 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\") pod \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " Nov 02 15:44:38.469492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.469485 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\" (UniqueName: \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\") pod \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\" (UID: \"1cd77af1-9903-4d33-89e5-b7ad9590ff61\") " Nov 02 15:44:38.469665 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.469642 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca" pod "1cd77af1-9903-4d33-89e5-b7ad9590ff61" (UID: "1cd77af1-9903-4d33-89e5-b7ad9590ff61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:38.469821 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.469785 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:38.471376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.471350 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9" pod "1cd77af1-9903-4d33-89e5-b7ad9590ff61" (UID: "1cd77af1-9903-4d33-89e5-b7ad9590ff61"). InnerVolumeSpecName "kube-api-access-szfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:38.471439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.471391 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca" pod "1cd77af1-9903-4d33-89e5-b7ad9590ff61" (UID: "1cd77af1-9903-4d33-89e5-b7ad9590ff61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:38.570969 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.570891 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szfb9\" (UniqueName: \"kubernetes.io/projected/1cd77af1-9903-4d33-89e5-b7ad9590ff61-kube-api-access-szfb9\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:38.571250 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.571222 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd77af1-9903-4d33-89e5-b7ad9590ff61-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:38.766895 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:38.766860 4469 scope.go:122] "RemoveContainer" containerID="b5d92c4ab72e3269e3d9d5ee5ba1e87c3c5c2cf01cc6f1581a1f6e6a55cacc70" Nov 02 15:44:39.315533 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:39.315505 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" Nov 02 15:44:39.315704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:39.315535 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb" event={"ID":"1cd77af1-9903-4d33-89e5-b7ad9590ff61","Type":"ContainerDied","Data":"9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1"} Nov 02 15:44:39.315704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:39.315579 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a49542d2943d4d3b277a16e770d69f20e4b435ab5b4566f4927e0410fa000e1" Nov 02 15:44:47.571497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.571460 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n"] Nov 02 15:44:47.572007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.571962 4469 cpu_manager.go:414] "containerMap: removing container" podUID="1cd77af1-9903-4d33-89e5-b7ad9590ff61" containerName="revoke-keys" Nov 02 15:44:47.572007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.571976 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd77af1-9903-4d33-89e5-b7ad9590ff61" containerName="revoke-keys" Nov 02 15:44:47.572118 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.572085 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd77af1-9903-4d33-89e5-b7ad9590ff61" containerName="revoke-keys" Nov 02 15:44:47.576629 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.576609 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.631573 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.631547 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:44:47.631760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.631591 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:44:47.665764 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.665726 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmpn\" (UniqueName: \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.665978 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.665949 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.666043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.665998 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.679196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.679171 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:44:47.682067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.682043 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n"] Nov 02 15:44:47.729939 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.729907 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:44:47.766960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.766921 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.767121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.766970 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.767121 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.767087 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmpn\" (UniqueName: \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.767619 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.767597 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.769462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.769433 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.829269 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.829190 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmpn\" (UniqueName: \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\") pod \"maas-api-revoke-sub-ec843018-8vv4n\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:47.890816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:47.890770 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:48.264896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:48.264628 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n"] Nov 02 15:44:48.267404 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:44:48.267378 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020367fb_e237_4dbd_87de_050c1697ccc5.slice/crio-27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9 WatchSource:0}: Error finding container 27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9: Status 404 returned error can't find the container with id 27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9 Nov 02 15:44:48.360174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:48.360140 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" event={"ID":"020367fb-e237-4dbd-87de-050c1697ccc5","Type":"ContainerStarted","Data":"27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9"} Nov 02 15:44:49.366915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:49.366824 4469 generic.go:356] "Generic (PLEG): container finished" podID="020367fb-e237-4dbd-87de-050c1697ccc5" containerID="137f0d117c9438d4559209f1c6246f4a0178a278d938cba6067490f14dd5c819" exitCode=0 Nov 02 15:44:49.367295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:49.366911 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" event={"ID":"020367fb-e237-4dbd-87de-050c1697ccc5","Type":"ContainerDied","Data":"137f0d117c9438d4559209f1c6246f4a0178a278d938cba6067490f14dd5c819"} Nov 02 15:44:49.805397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:49.805359 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:50.509909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.509883 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:50.595459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.595426 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\" (UniqueName: \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\") pod \"020367fb-e237-4dbd-87de-050c1697ccc5\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " Nov 02 15:44:50.595626 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.595590 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\") pod \"020367fb-e237-4dbd-87de-050c1697ccc5\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " Nov 02 15:44:50.595679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.595644 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\") pod \"020367fb-e237-4dbd-87de-050c1697ccc5\" (UID: \"020367fb-e237-4dbd-87de-050c1697ccc5\") " Nov 02 15:44:50.596037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.596009 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca" pod "020367fb-e237-4dbd-87de-050c1697ccc5" (UID: "020367fb-e237-4dbd-87de-050c1697ccc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:50.597542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.597513 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn" pod "020367fb-e237-4dbd-87de-050c1697ccc5" (UID: "020367fb-e237-4dbd-87de-050c1697ccc5"). InnerVolumeSpecName "kube-api-access-kqmpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:50.597622 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.597564 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca" pod "020367fb-e237-4dbd-87de-050c1697ccc5" (UID: "020367fb-e237-4dbd-87de-050c1697ccc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:50.696711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.696629 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmpn\" (UniqueName: \"kubernetes.io/projected/020367fb-e237-4dbd-87de-050c1697ccc5-kube-api-access-kqmpn\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:50.696711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.696658 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:50.696711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:50.696669 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/020367fb-e237-4dbd-87de-050c1697ccc5-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:44:51.378148 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:51.378117 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" Nov 02 15:44:51.378293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:51.378157 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n" event={"ID":"020367fb-e237-4dbd-87de-050c1697ccc5","Type":"ContainerDied","Data":"27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9"} Nov 02 15:44:51.378293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:51.378210 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27692010aa7cd7558bc33bb019126e69f7c486645e212d7f8e149ca81c2d15a9" Nov 02 15:44:51.681452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:51.681378 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:44:58.380840 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:58.380789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:58.625172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:58.625101 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:59.696125 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:44:59.696091 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:00.170731 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.170700 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:00.171129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.171116 4469 cpu_manager.go:414] "containerMap: removing container" podUID="020367fb-e237-4dbd-87de-050c1697ccc5" containerName="revoke-keys" Nov 02 15:45:00.171178 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.171131 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020367fb-e237-4dbd-87de-050c1697ccc5" containerName="revoke-keys" Nov 02 15:45:00.171219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.171203 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="020367fb-e237-4dbd-87de-050c1697ccc5" containerName="revoke-keys" Nov 02 15:45:00.174419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.174400 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.228429 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.228402 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:45:00.277504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.277470 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:45:00.277504 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.277505 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:45:00.279621 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.279601 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:00.294876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.294848 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.295077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.295052 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzss\" (UniqueName: \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.396520 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.396488 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzss\" (UniqueName: \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.396725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.396582 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.397280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.397252 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.458096 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.458023 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzss\" (UniqueName: \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\") pod \"maas-api-key-cleanup-29893905-n6wjj\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.486179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.486151 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:00.858376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:00.858343 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:00.861004 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:45:00.860977 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d2e2dc_8122_4254_a9ea_4bba573a0532.slice/crio-136abe2663e630dea35fb1a4bc900a1c0a9a56646c4d14ff1b5b2bf659d0613e WatchSource:0}: Error finding container 136abe2663e630dea35fb1a4bc900a1c0a9a56646c4d14ff1b5b2bf659d0613e: Status 404 returned error can't find the container with id 136abe2663e630dea35fb1a4bc900a1c0a9a56646c4d14ff1b5b2bf659d0613e Nov 02 15:45:01.426773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:01.426734 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerStarted","Data":"4a9a6eca02c23e2a57e31560d861c1875e9b14a29250b72dc5b0b02dae342be4"} Nov 02 15:45:01.426958 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:01.426793 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerStarted","Data":"136abe2663e630dea35fb1a4bc900a1c0a9a56646c4d14ff1b5b2bf659d0613e"} Nov 02 15:45:01.544019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:01.543788 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" podStartSLOduration=1.543774026 podStartE2EDuration="1.543774026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:01.542794448 +0000 UTC m=+2904.426396969" watchObservedRunningTime="2026-11-02 15:45:01.543774026 +0000 UTC m=+2904.427376545" Nov 02 15:45:02.433130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:02.433093 4469 generic.go:356] "Generic (PLEG): container finished" podID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerID="4a9a6eca02c23e2a57e31560d861c1875e9b14a29250b72dc5b0b02dae342be4" exitCode=60 Nov 02 15:45:02.433532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:02.433133 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerDied","Data":"4a9a6eca02c23e2a57e31560d861c1875e9b14a29250b72dc5b0b02dae342be4"} Nov 02 15:45:02.433532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:02.433390 4469 scope.go:122] "RemoveContainer" containerID="4a9a6eca02c23e2a57e31560d861c1875e9b14a29250b72dc5b0b02dae342be4" Nov 02 15:45:03.440636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:03.440605 4469 generic.go:356] "Generic (PLEG): container finished" podID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerID="c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156" exitCode=60 Nov 02 15:45:03.441108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:03.440663 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerDied","Data":"c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156"} Nov 02 15:45:03.441108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:03.440709 4469 scope.go:122] "RemoveContainer" containerID="4a9a6eca02c23e2a57e31560d861c1875e9b14a29250b72dc5b0b02dae342be4" Nov 02 15:45:03.441108 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:03.440952 4469 scope.go:122] "RemoveContainer" containerID="c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156" Nov 02 15:45:03.441228 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:45:03.441194 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-n6wjj_odh-ai-gateway-infra(52d2e2dc-8122-4254-a9ea-4bba573a0532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" Nov 02 15:45:04.448552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:04.448522 4469 scope.go:122] "RemoveContainer" containerID="c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156" Nov 02 15:45:04.449096 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:45:04.448747 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-n6wjj_odh-ai-gateway-infra(52d2e2dc-8122-4254-a9ea-4bba573a0532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" Nov 02 15:45:04.885576 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:04.885541 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:45:06.237823 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:45:06.236111 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:45:14.231816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:14.231769 4469 scope.go:122] "RemoveContainer" containerID="c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156" Nov 02 15:45:14.526267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:14.526201 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerStarted","Data":"e13751836ac76283de677670b4fc714c6b0086df558f8f894d44f4169d0acbfd"} Nov 02 15:45:15.533421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:15.533378 4469 generic.go:356] "Generic (PLEG): container finished" podID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerID="e13751836ac76283de677670b4fc714c6b0086df558f8f894d44f4169d0acbfd" exitCode=60 Nov 02 15:45:15.533937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:15.533455 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerDied","Data":"e13751836ac76283de677670b4fc714c6b0086df558f8f894d44f4169d0acbfd"} Nov 02 15:45:15.533937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:15.533533 4469 scope.go:122] "RemoveContainer" containerID="c84cb38b1b1fd54ffb36e01b02bebb2fbb92f6d6aaf843d2f4120b264c7c1156" Nov 02 15:45:15.533937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:15.533918 4469 scope.go:122] "RemoveContainer" containerID="e13751836ac76283de677670b4fc714c6b0086df558f8f894d44f4169d0acbfd" Nov 02 15:45:15.534176 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:45:15.534157 4469 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-n6wjj_odh-ai-gateway-infra(52d2e2dc-8122-4254-a9ea-4bba573a0532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" Nov 02 15:45:15.656944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:15.656908 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:16.675762 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.675738 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:16.784922 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.784890 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\") pod \"52d2e2dc-8122-4254-a9ea-4bba573a0532\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " Nov 02 15:45:16.785088 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.785000 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\" (UniqueName: \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\") pod \"52d2e2dc-8122-4254-a9ea-4bba573a0532\" (UID: \"52d2e2dc-8122-4254-a9ea-4bba573a0532\") " Nov 02 15:45:16.785291 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.785270 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca" pod "52d2e2dc-8122-4254-a9ea-4bba573a0532" (UID: "52d2e2dc-8122-4254-a9ea-4bba573a0532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:16.785352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.785338 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d2e2dc-8122-4254-a9ea-4bba573a0532-maas-api-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:16.786997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.786971 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss" pod "52d2e2dc-8122-4254-a9ea-4bba573a0532" (UID: "52d2e2dc-8122-4254-a9ea-4bba573a0532"). InnerVolumeSpecName "kube-api-access-6jzss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:16.885900 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:16.885868 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzss\" (UniqueName: \"kubernetes.io/projected/52d2e2dc-8122-4254-a9ea-4bba573a0532-kube-api-access-6jzss\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.547066 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:17.546984 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" Nov 02 15:45:17.547066 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:17.547025 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj" event={"ID":"52d2e2dc-8122-4254-a9ea-4bba573a0532","Type":"ContainerDied","Data":"136abe2663e630dea35fb1a4bc900a1c0a9a56646c4d14ff1b5b2bf659d0613e"} Nov 02 15:45:17.547287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:17.547090 4469 scope.go:122] "RemoveContainer" containerID="e13751836ac76283de677670b4fc714c6b0086df558f8f894d44f4169d0acbfd" Nov 02 15:45:17.716320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:17.716285 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:17.719021 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:17.718991 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-n6wjj"] Nov 02 15:45:19.237760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:19.237725 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" path="/var/lib/kubelet/pods/52d2e2dc-8122-4254-a9ea-4bba573a0532/volumes" Nov 02 15:45:20.251381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:20.251349 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:23.169985 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.169955 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz"] Nov 02 15:45:23.170425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170411 4469 cpu_manager.go:414] "containerMap: removing container" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170427 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170440 4469 cpu_manager.go:414] "containerMap: removing container" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170445 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170470 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170468 4469 cpu_manager.go:414] "containerMap: removing container" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170475 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170541 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.170590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.170551 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:23.174747 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.174731 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.240938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.240913 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:45:23.241260 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.241244 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:23.245347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.245317 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz"] Nov 02 15:45:23.248202 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.248171 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.248319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.248205 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.248386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.248327 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjwf\" (UniqueName: \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.290545 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.290520 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:45:23.290545 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.290520 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:45:23.348901 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.348874 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.349031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.348916 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.349031 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.348988 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjwf\" (UniqueName: \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.349513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.349494 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.351154 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.351134 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.416724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.416700 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjwf\" (UniqueName: \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\") pod \"maas-api-revoke-sub-046e1343-6stqz\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.487547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.487526 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:23.559810 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.559767 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:23.859700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:23.859615 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz"] Nov 02 15:45:23.862535 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:45:23.862507 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b895d5_f307_45cd_aca4_1f9b6c0038fe.slice/crio-202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de WatchSource:0}: Error finding container 202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de: Status 404 returned error can't find the container with id 202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de Nov 02 15:45:24.584156 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:24.584124 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" event={"ID":"b5b895d5-f307-45cd-aca4-1f9b6c0038fe","Type":"ContainerStarted","Data":"6269fd4b2d169131ebcccbeb733cb612459e8df2b69f4dda790606faa829621d"} Nov 02 15:45:24.584538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:24.584180 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" event={"ID":"b5b895d5-f307-45cd-aca4-1f9b6c0038fe","Type":"ContainerStarted","Data":"202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de"} Nov 02 15:45:24.706423 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:24.706374 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" podStartSLOduration=1.706358223 podStartE2EDuration="1.706358223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:24.704679067 +0000 UTC m=+2927.588281589" watchObservedRunningTime="2026-11-02 15:45:24.706358223 +0000 UTC m=+2927.589960742" Nov 02 15:45:25.069615 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:25.069578 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:25.275348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:25.275322 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:25.590956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:25.590927 4469 generic.go:356] "Generic (PLEG): container finished" podID="b5b895d5-f307-45cd-aca4-1f9b6c0038fe" containerID="6269fd4b2d169131ebcccbeb733cb612459e8df2b69f4dda790606faa829621d" exitCode=0 Nov 02 15:45:25.591341 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:25.591006 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" event={"ID":"b5b895d5-f307-45cd-aca4-1f9b6c0038fe","Type":"ContainerDied","Data":"6269fd4b2d169131ebcccbeb733cb612459e8df2b69f4dda790606faa829621d"} Nov 02 15:45:26.727001 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.726976 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:26.783200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.783168 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\" (UniqueName: \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\") pod \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " Nov 02 15:45:26.783368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.783268 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\") pod \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " Nov 02 15:45:26.783368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.783315 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\") pod \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\" (UID: \"b5b895d5-f307-45cd-aca4-1f9b6c0038fe\") " Nov 02 15:45:26.783715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.783689 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca" pod "b5b895d5-f307-45cd-aca4-1f9b6c0038fe" (UID: "b5b895d5-f307-45cd-aca4-1f9b6c0038fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:26.785325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.785301 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca" pod "b5b895d5-f307-45cd-aca4-1f9b6c0038fe" (UID: "b5b895d5-f307-45cd-aca4-1f9b6c0038fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:26.785391 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.785344 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf" pod "b5b895d5-f307-45cd-aca4-1f9b6c0038fe" (UID: "b5b895d5-f307-45cd-aca4-1f9b6c0038fe"). InnerVolumeSpecName "kube-api-access-lbjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:26.884660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.884591 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbjwf\" (UniqueName: \"kubernetes.io/projected/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-kube-api-access-lbjwf\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:26.884660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.884620 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:26.884660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:26.884630 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b895d5-f307-45cd-aca4-1f9b6c0038fe-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:27.601401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:27.601324 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" Nov 02 15:45:27.601401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:27.601353 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz" event={"ID":"b5b895d5-f307-45cd-aca4-1f9b6c0038fe","Type":"ContainerDied","Data":"202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de"} Nov 02 15:45:27.601401 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:27.601400 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="202755ae64c6f4d951b4b8caa6e550ba98fc589d07fcdf9eaa599b02724965de" Nov 02 15:45:34.006908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.006873 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f"] Nov 02 15:45:34.007384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.007281 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b5b895d5-f307-45cd-aca4-1f9b6c0038fe" containerName="revoke-keys" Nov 02 15:45:34.007384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.007293 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b895d5-f307-45cd-aca4-1f9b6c0038fe" containerName="revoke-keys" Nov 02 15:45:34.007384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.007365 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b895d5-f307-45cd-aca4-1f9b6c0038fe" containerName="revoke-keys" Nov 02 15:45:34.007384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.007377 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d2e2dc-8122-4254-a9ea-4bba573a0532" containerName="cleanup" Nov 02 15:45:34.010853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.010832 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.062859 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.062825 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:45:34.112347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.112312 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:45:34.112347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.112316 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:34.112543 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.112379 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:45:34.114514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.114495 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f"] Nov 02 15:45:34.153305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.153271 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.153305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.153307 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.153574 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.153338 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv28t\" (UniqueName: \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.254408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.254368 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.254408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.254415 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.254613 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.254448 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv28t\" (UniqueName: \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.255129 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.255093 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.256616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.256595 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.317730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.317668 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv28t\" (UniqueName: \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\") pod \"maas-api-revoke-sub-d3fba056-z4t4f\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.323759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.323735 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:34.696570 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:34.696543 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f"] Nov 02 15:45:34.699017 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:45:34.698988 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2feb2494_c7b8_4f7e_aa13_89efa6d80de7.slice/crio-f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa WatchSource:0}: Error finding container f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa: Status 404 returned error can't find the container with id f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa Nov 02 15:45:35.641093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:35.641061 4469 generic.go:356] "Generic (PLEG): container finished" podID="2feb2494-c7b8-4f7e-aa13-89efa6d80de7" containerID="9ba0f0211124ff5e8316568eee6c6390608e7587d55dd280ccab12ca00355811" exitCode=0 Nov 02 15:45:35.641492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:35.641155 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" event={"ID":"2feb2494-c7b8-4f7e-aa13-89efa6d80de7","Type":"ContainerDied","Data":"9ba0f0211124ff5e8316568eee6c6390608e7587d55dd280ccab12ca00355811"} Nov 02 15:45:35.641492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:35.641219 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" event={"ID":"2feb2494-c7b8-4f7e-aa13-89efa6d80de7","Type":"ContainerStarted","Data":"f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa"} Nov 02 15:45:36.048618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.048578 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:45:36.189568 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:45:36.189520 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:45:36.812816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.812779 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:36.880971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.880939 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\" (UniqueName: \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\") pod \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " Nov 02 15:45:36.881100 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.881063 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\") pod \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " Nov 02 15:45:36.881155 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.881105 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\") pod \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\" (UID: \"2feb2494-c7b8-4f7e-aa13-89efa6d80de7\") " Nov 02 15:45:36.881514 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.881483 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca" pod "2feb2494-c7b8-4f7e-aa13-89efa6d80de7" (UID: "2feb2494-c7b8-4f7e-aa13-89efa6d80de7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:36.883638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.883611 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca" pod "2feb2494-c7b8-4f7e-aa13-89efa6d80de7" (UID: "2feb2494-c7b8-4f7e-aa13-89efa6d80de7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:36.887384 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.887358 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t" pod "2feb2494-c7b8-4f7e-aa13-89efa6d80de7" (UID: "2feb2494-c7b8-4f7e-aa13-89efa6d80de7"). InnerVolumeSpecName "kube-api-access-pv28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:36.982686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.982633 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv28t\" (UniqueName: \"kubernetes.io/projected/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-kube-api-access-pv28t\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.982686 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.982679 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.982986 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:36.982699 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2feb2494-c7b8-4f7e-aa13-89efa6d80de7-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:45:37.668174 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:37.668084 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" event={"ID":"2feb2494-c7b8-4f7e-aa13-89efa6d80de7","Type":"ContainerDied","Data":"f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa"} Nov 02 15:45:37.668542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:37.668520 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9946830964af34d086a8c009facd0340fa6c20aa551301a9be5fa31052dbaaa" Nov 02 15:45:37.668721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:37.668707 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f" Nov 02 15:45:38.791823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:38.790778 4469 scope.go:122] "RemoveContainer" containerID="2f063bfdf51e25461bf69cf8e5d25dbf55f2610ab0d9757e1840f137c44284ea" Nov 02 15:45:38.805194 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:38.805172 4469 scope.go:122] "RemoveContainer" containerID="40e9136d62aa428af64aa862b7e63bcb9894469391ec8f68730c0f61130cc37b" Nov 02 15:45:38.813715 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:38.813694 4469 scope.go:122] "RemoveContainer" containerID="bc57018fef2421dc37f39b168f86f9a678413acc54bd2af3037bb048d398cf33" Nov 02 15:45:46.768696 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:46.768662 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:45:50.103130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:50.103091 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:59.272085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:45:59.272052 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:46:00.063057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:00.063003 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:06.185612 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:46:06.185578 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:46:08.799042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.799002 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n"] Nov 02 15:46:08.799680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.799632 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2feb2494-c7b8-4f7e-aa13-89efa6d80de7" containerName="revoke-keys" Nov 02 15:46:08.799680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.799649 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feb2494-c7b8-4f7e-aa13-89efa6d80de7" containerName="revoke-keys" Nov 02 15:46:08.799835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.799791 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feb2494-c7b8-4f7e-aa13-89efa6d80de7" containerName="revoke-keys" Nov 02 15:46:08.803085 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.803062 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.817625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.817595 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.817775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.817661 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd452\" (UniqueName: \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.817775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.817756 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.856728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.856691 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:46:08.856728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.856721 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:46:08.856966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.856851 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:46:08.897642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.897614 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:46:08.908473 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.908444 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n"] Nov 02 15:46:08.918830 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.918791 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.918920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.918896 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.918960 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.918944 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd452\" (UniqueName: \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.919624 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.919602 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.921185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.921165 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:08.981305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:08.981282 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd452\" (UniqueName: \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\") pod \"maas-api-revoke-sub-ae594979-p9z8n\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:09.116546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:09.116483 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:09.486081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:09.486049 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n"] Nov 02 15:46:09.488004 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:46:09.487976 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa37ca25_8172_4419_8ca5_835a9b49b87b.slice/crio-cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56 WatchSource:0}: Error finding container cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56: Status 404 returned error can't find the container with id cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56 Nov 02 15:46:09.870134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:09.870015 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" event={"ID":"fa37ca25-8172-4419-8ca5-835a9b49b87b","Type":"ContainerStarted","Data":"932ca2ec60b0006f381cef59bd7b7c93e30ec249c1ae4af16a15fc43075b811a"} Nov 02 15:46:09.870134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:09.870112 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" event={"ID":"fa37ca25-8172-4419-8ca5-835a9b49b87b","Type":"ContainerStarted","Data":"cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56"} Nov 02 15:46:09.988998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:09.988941 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" podStartSLOduration=1.9889269120000002 podStartE2EDuration="1.988926912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:09.986377889 +0000 UTC m=+2972.869980408" watchObservedRunningTime="2026-11-02 15:46:09.988926912 +0000 UTC m=+2972.872529432" Nov 02 15:46:10.876968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:10.876937 4469 generic.go:356] "Generic (PLEG): container finished" podID="fa37ca25-8172-4419-8ca5-835a9b49b87b" containerID="932ca2ec60b0006f381cef59bd7b7c93e30ec249c1ae4af16a15fc43075b811a" exitCode=0 Nov 02 15:46:10.876968 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:10.876975 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" event={"ID":"fa37ca25-8172-4419-8ca5-835a9b49b87b","Type":"ContainerDied","Data":"932ca2ec60b0006f381cef59bd7b7c93e30ec249c1ae4af16a15fc43075b811a"} Nov 02 15:46:12.011822 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.011776 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:12.047678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.047654 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\" (UniqueName: \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\") pod \"fa37ca25-8172-4419-8ca5-835a9b49b87b\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " Nov 02 15:46:12.047831 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.047792 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\") pod \"fa37ca25-8172-4419-8ca5-835a9b49b87b\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " Nov 02 15:46:12.047923 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.047905 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\") pod \"fa37ca25-8172-4419-8ca5-835a9b49b87b\" (UID: \"fa37ca25-8172-4419-8ca5-835a9b49b87b\") " Nov 02 15:46:12.048255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.048232 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca" pod "fa37ca25-8172-4419-8ca5-835a9b49b87b" (UID: "fa37ca25-8172-4419-8ca5-835a9b49b87b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:12.049767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.049732 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca" pod "fa37ca25-8172-4419-8ca5-835a9b49b87b" (UID: "fa37ca25-8172-4419-8ca5-835a9b49b87b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:12.049875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.049786 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452" pod "fa37ca25-8172-4419-8ca5-835a9b49b87b" (UID: "fa37ca25-8172-4419-8ca5-835a9b49b87b"). InnerVolumeSpecName "kube-api-access-wd452". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:12.148955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.148874 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd452\" (UniqueName: \"kubernetes.io/projected/fa37ca25-8172-4419-8ca5-835a9b49b87b-kube-api-access-wd452\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:12.148955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.148918 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:12.148955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.148936 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa37ca25-8172-4419-8ca5-835a9b49b87b-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:12.208315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.208283 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:12.888157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.888127 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" Nov 02 15:46:12.888319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.888152 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n" event={"ID":"fa37ca25-8172-4419-8ca5-835a9b49b87b","Type":"ContainerDied","Data":"cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56"} Nov 02 15:46:12.888319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:12.888206 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9ab926fc2bd6c69cc846723f9b8263ec9153faddfa662725d912a6f5825b56" Nov 02 15:46:21.667321 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:21.667284 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:46:23.942528 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:23.942489 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:31.166130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:31.166092 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:46:36.185430 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:46:36.185397 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:46:37.000425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.000378 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z"] Nov 02 15:46:37.000956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.000934 4469 cpu_manager.go:414] "containerMap: removing container" podUID="fa37ca25-8172-4419-8ca5-835a9b49b87b" containerName="revoke-keys" Nov 02 15:46:37.001076 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.000957 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa37ca25-8172-4419-8ca5-835a9b49b87b" containerName="revoke-keys" Nov 02 15:46:37.001150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.001100 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa37ca25-8172-4419-8ca5-835a9b49b87b" containerName="revoke-keys" Nov 02 15:46:37.004409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.004387 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.100593 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.100555 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxprk\" (UniqueName: \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.100748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.100677 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.100748 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.100733 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.106330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.106309 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:46:37.106457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.106339 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:46:37.106457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.106346 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:46:37.106457 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.106360 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:46:37.108494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.108476 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z"] Nov 02 15:46:37.201390 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.201355 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.201882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.201405 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.201882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.201455 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxprk\" (UniqueName: \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.202081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.202061 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.203741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.203717 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.311373 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.311299 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:46:37.323071 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.323045 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxprk\" (UniqueName: \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\") pod \"maas-api-revoke-sub-a4aef7d7-8qt8z\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:37.669067 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.668988 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:46:37.677758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:37.677739 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:38.060060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:38.060027 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z"] Nov 02 15:46:38.063031 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:46:38.063004 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fcdc4a_697d_4e91_815e_25c6ab054313.slice/crio-b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242 WatchSource:0}: Error finding container b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242: Status 404 returned error can't find the container with id b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242 Nov 02 15:46:39.030974 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:39.030944 4469 generic.go:356] "Generic (PLEG): container finished" podID="c0fcdc4a-697d-4e91-815e-25c6ab054313" containerID="b96f317d0cbc4183b1c9e0882c260ed1d980ee44d283aa43f2f381087c27d386" exitCode=0 Nov 02 15:46:39.031365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:39.030994 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" event={"ID":"c0fcdc4a-697d-4e91-815e-25c6ab054313","Type":"ContainerDied","Data":"b96f317d0cbc4183b1c9e0882c260ed1d980ee44d283aa43f2f381087c27d386"} Nov 02 15:46:39.031365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:39.031036 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" event={"ID":"c0fcdc4a-697d-4e91-815e-25c6ab054313","Type":"ContainerStarted","Data":"b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242"} Nov 02 15:46:40.168465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.168437 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:40.231776 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.231750 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\" (UniqueName: \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\") pod \"c0fcdc4a-697d-4e91-815e-25c6ab054313\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " Nov 02 15:46:40.231956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.231787 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\") pod \"c0fcdc4a-697d-4e91-815e-25c6ab054313\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " Nov 02 15:46:40.231956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.231888 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\") pod \"c0fcdc4a-697d-4e91-815e-25c6ab054313\" (UID: \"c0fcdc4a-697d-4e91-815e-25c6ab054313\") " Nov 02 15:46:40.232159 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.232133 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca" pod "c0fcdc4a-697d-4e91-815e-25c6ab054313" (UID: "c0fcdc4a-697d-4e91-815e-25c6ab054313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.233683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.233658 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk" pod "c0fcdc4a-697d-4e91-815e-25c6ab054313" (UID: "c0fcdc4a-697d-4e91-815e-25c6ab054313"). InnerVolumeSpecName "kube-api-access-fxprk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.233839 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.233818 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca" pod "c0fcdc4a-697d-4e91-815e-25c6ab054313" (UID: "c0fcdc4a-697d-4e91-815e-25c6ab054313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:40.333787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.333709 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxprk\" (UniqueName: \"kubernetes.io/projected/c0fcdc4a-697d-4e91-815e-25c6ab054313-kube-api-access-fxprk\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.333787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.333734 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.333787 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:40.333745 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fcdc4a-697d-4e91-815e-25c6ab054313-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.041485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:41.041456 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" Nov 02 15:46:41.041485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:41.041470 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z" event={"ID":"c0fcdc4a-697d-4e91-815e-25c6ab054313","Type":"ContainerDied","Data":"b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242"} Nov 02 15:46:41.041698 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:41.041513 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d754a0475066e46d18516d191aa62a86ff359d4c68c09045a81618dae92242" Nov 02 15:46:49.884937 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:49.884848 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:46:50.453442 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:46:50.453393 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:00.346950 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:00.346916 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:02.180289 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:02.180256 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:47:06.186145 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:47:06.186111 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:47:07.958549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:07.958509 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx"] Nov 02 15:47:07.959118 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:07.959096 4469 cpu_manager.go:414] "containerMap: removing container" podUID="c0fcdc4a-697d-4e91-815e-25c6ab054313" containerName="revoke-keys" Nov 02 15:47:07.959203 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:07.959120 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fcdc4a-697d-4e91-815e-25c6ab054313" containerName="revoke-keys" Nov 02 15:47:07.959275 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:07.959262 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fcdc4a-697d-4e91-815e-25c6ab054313" containerName="revoke-keys" Nov 02 15:47:07.962525 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:07.962503 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.012081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.012048 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.012205 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.012124 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.012254 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.012222 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scv4\" (UniqueName: \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.014863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.014846 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:47:08.064115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.064072 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:08.064115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.064116 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:47:08.064325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.064245 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:47:08.067639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.067617 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx"] Nov 02 15:47:08.113235 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.113206 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scv4\" (UniqueName: \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.113400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.113256 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.113400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.113321 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.114080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.114061 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.115717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.115696 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.174350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.174327 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scv4\" (UniqueName: \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\") pod \"maas-api-revoke-sub-79805ca9-tlbsx\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.275661 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.275633 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:08.650359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:08.650299 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx"] Nov 02 15:47:08.653442 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:47:08.653407 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2027236e_5c94_4b25_8128_60e13e7bea98.slice/crio-2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f WatchSource:0}: Error finding container 2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f: Status 404 returned error can't find the container with id 2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f Nov 02 15:47:09.199150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:09.199067 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" event={"ID":"2027236e-5c94-4b25-8128-60e13e7bea98","Type":"ContainerStarted","Data":"043cd86743dd015b0c4258add54715dcd2014733e63ab76e081627c1c241bb4b"} Nov 02 15:47:09.199150 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:09.199128 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" event={"ID":"2027236e-5c94-4b25-8128-60e13e7bea98","Type":"ContainerStarted","Data":"2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f"} Nov 02 15:47:09.327000 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:09.326946 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" podStartSLOduration=2.326931617 podStartE2EDuration="2.326931617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:09.324977342 +0000 UTC m=+3032.208579862" watchObservedRunningTime="2026-11-02 15:47:09.326931617 +0000 UTC m=+3032.210534137" Nov 02 15:47:10.205464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:10.205430 4469 generic.go:356] "Generic (PLEG): container finished" podID="2027236e-5c94-4b25-8128-60e13e7bea98" containerID="043cd86743dd015b0c4258add54715dcd2014733e63ab76e081627c1c241bb4b" exitCode=0 Nov 02 15:47:10.205874 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:10.205487 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" event={"ID":"2027236e-5c94-4b25-8128-60e13e7bea98","Type":"ContainerDied","Data":"043cd86743dd015b0c4258add54715dcd2014733e63ab76e081627c1c241bb4b"} Nov 02 15:47:11.344161 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.344138 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:11.445532 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.445500 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\") pod \"2027236e-5c94-4b25-8128-60e13e7bea98\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " Nov 02 15:47:11.445687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.445588 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\" (UniqueName: \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\") pod \"2027236e-5c94-4b25-8128-60e13e7bea98\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " Nov 02 15:47:11.445759 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.445695 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\") pod \"2027236e-5c94-4b25-8128-60e13e7bea98\" (UID: \"2027236e-5c94-4b25-8128-60e13e7bea98\") " Nov 02 15:47:11.446046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.446025 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca" pod "2027236e-5c94-4b25-8128-60e13e7bea98" (UID: "2027236e-5c94-4b25-8128-60e13e7bea98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:11.447476 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.447457 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca" pod "2027236e-5c94-4b25-8128-60e13e7bea98" (UID: "2027236e-5c94-4b25-8128-60e13e7bea98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:11.447537 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.447521 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4" pod "2027236e-5c94-4b25-8128-60e13e7bea98" (UID: "2027236e-5c94-4b25-8128-60e13e7bea98"). InnerVolumeSpecName "kube-api-access-5scv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:11.546721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.546647 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:11.546721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.546673 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scv4\" (UniqueName: \"kubernetes.io/projected/2027236e-5c94-4b25-8128-60e13e7bea98-kube-api-access-5scv4\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:11.546721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:11.546683 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2027236e-5c94-4b25-8128-60e13e7bea98-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:12.216097 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:12.216065 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" Nov 02 15:47:12.216279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:12.216113 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx" event={"ID":"2027236e-5c94-4b25-8128-60e13e7bea98","Type":"ContainerDied","Data":"2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f"} Nov 02 15:47:12.216279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:12.216161 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2060ad3b02cc140561ae425693ed018f844b2111c18eb72bd61efdbde6bcb44f" Nov 02 15:47:22.640355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:22.640315 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:47:24.381987 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:24.381947 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:36.186929 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:47:36.186894 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:47:38.759944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.759911 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf"] Nov 02 15:47:38.760360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.760331 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2027236e-5c94-4b25-8128-60e13e7bea98" containerName="revoke-keys" Nov 02 15:47:38.760360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.760343 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2027236e-5c94-4b25-8128-60e13e7bea98" containerName="revoke-keys" Nov 02 15:47:38.760459 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.760447 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2027236e-5c94-4b25-8128-60e13e7bea98" containerName="revoke-keys" Nov 02 15:47:38.764099 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.764076 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.823336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.823308 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:38.838060 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.838026 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.838190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.838151 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.838285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.838265 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bdg\" (UniqueName: \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.868979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.868946 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:47:38.869131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.868991 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:47:38.869131 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.869033 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:47:38.871152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.871133 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf"] Nov 02 15:47:38.938973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.938947 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bdg\" (UniqueName: \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.939101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.938994 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.939101 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.939033 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.939627 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.939606 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.941287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.941268 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:38.999914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:38.999884 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bdg\" (UniqueName: \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\") pod \"maas-api-revoke-sub-0b7e8edb-lxfkf\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:39.076081 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:39.076014 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:39.463230 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:39.463202 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf"] Nov 02 15:47:39.466076 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:47:39.466049 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e686eab_236c_41a0_8cb1_3172ea326a92.slice/crio-30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d WatchSource:0}: Error finding container 30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d: Status 404 returned error can't find the container with id 30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d Nov 02 15:47:40.371354 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:40.371320 4469 generic.go:356] "Generic (PLEG): container finished" podID="2e686eab-236c-41a0-8cb1-3172ea326a92" containerID="485bcb6a4d7efd5c8ca3834e456685f04c6ccfc35eb2c06dc65500d9520cda10" exitCode=0 Nov 02 15:47:40.371774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:40.371383 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" event={"ID":"2e686eab-236c-41a0-8cb1-3172ea326a92","Type":"ContainerDied","Data":"485bcb6a4d7efd5c8ca3834e456685f04c6ccfc35eb2c06dc65500d9520cda10"} Nov 02 15:47:40.371774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:40.371419 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" event={"ID":"2e686eab-236c-41a0-8cb1-3172ea326a92","Type":"ContainerStarted","Data":"30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d"} Nov 02 15:47:41.505976 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.505954 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:41.567535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.567510 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\") pod \"2e686eab-236c-41a0-8cb1-3172ea326a92\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " Nov 02 15:47:41.567684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.567610 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\") pod \"2e686eab-236c-41a0-8cb1-3172ea326a92\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " Nov 02 15:47:41.567684 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.567659 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\" (UniqueName: \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\") pod \"2e686eab-236c-41a0-8cb1-3172ea326a92\" (UID: \"2e686eab-236c-41a0-8cb1-3172ea326a92\") " Nov 02 15:47:41.567980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.567954 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca" pod "2e686eab-236c-41a0-8cb1-3172ea326a92" (UID: "2e686eab-236c-41a0-8cb1-3172ea326a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:41.569725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.569703 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca" pod "2e686eab-236c-41a0-8cb1-3172ea326a92" (UID: "2e686eab-236c-41a0-8cb1-3172ea326a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:41.569725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.569715 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg" pod "2e686eab-236c-41a0-8cb1-3172ea326a92" (UID: "2e686eab-236c-41a0-8cb1-3172ea326a92"). InnerVolumeSpecName "kube-api-access-g2bdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:41.669370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.669299 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bdg\" (UniqueName: \"kubernetes.io/projected/2e686eab-236c-41a0-8cb1-3172ea326a92-kube-api-access-g2bdg\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:41.669370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.669328 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:41.669370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:41.669342 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e686eab-236c-41a0-8cb1-3172ea326a92-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:42.382733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:42.382692 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" event={"ID":"2e686eab-236c-41a0-8cb1-3172ea326a92","Type":"ContainerDied","Data":"30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d"} Nov 02 15:47:42.383006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:42.382744 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a6884ef2e712ec25b003a915020bec268457146665c09c21a585c5247c134d" Nov 02 15:47:42.383006 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:42.382711 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf" Nov 02 15:47:43.095218 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:43.095181 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:43.295527 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:43.295494 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:50.115871 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.115840 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv"] Nov 02 15:47:50.116293 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.116279 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2e686eab-236c-41a0-8cb1-3172ea326a92" containerName="revoke-keys" Nov 02 15:47:50.116350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.116293 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e686eab-236c-41a0-8cb1-3172ea326a92" containerName="revoke-keys" Nov 02 15:47:50.116386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.116369 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e686eab-236c-41a0-8cb1-3172ea326a92" containerName="revoke-keys" Nov 02 15:47:50.119638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.119622 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.172512 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.172480 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:47:50.172722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.172534 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:50.222133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.222092 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:47:50.222295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.222143 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:47:50.224416 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.224397 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv"] Nov 02 15:47:50.256449 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.256415 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.256632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.256457 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.256632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.256557 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7927\" (UniqueName: \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.357454 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.357416 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7927\" (UniqueName: \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.357662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.357557 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.357662 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.357604 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.358284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.358256 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.359725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.359701 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.418897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.418816 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7927\" (UniqueName: \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\") pod \"maas-api-revoke-sub-3d4156f1-6lgcv\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.432176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.432154 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:50.790967 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.790932 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:50.806852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:50.806828 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv"] Nov 02 15:47:50.808821 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:47:50.808777 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a43b30a_ac61_44d7_96b2_9925d08625df.slice/crio-bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540 WatchSource:0}: Error finding container bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540: Status 404 returned error can't find the container with id bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540 Nov 02 15:47:51.429996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:51.429895 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" event={"ID":"6a43b30a-ac61-44d7-96b2-9925d08625df","Type":"ContainerStarted","Data":"f60cf4a563c31cf70cb0503f7eca8d8a0b52b964ef9c9c38851842383477bd13"} Nov 02 15:47:51.429996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:51.429965 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" event={"ID":"6a43b30a-ac61-44d7-96b2-9925d08625df","Type":"ContainerStarted","Data":"bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540"} Nov 02 15:47:51.549149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:51.549092 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" podStartSLOduration=1.5490769439999998 podStartE2EDuration="1.549076944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:51.546916127 +0000 UTC m=+3074.430518646" watchObservedRunningTime="2026-11-02 15:47:51.549076944 +0000 UTC m=+3074.432679490" Nov 02 15:47:51.680893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:51.680786 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:47:52.451824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:52.448330 4469 generic.go:356] "Generic (PLEG): container finished" podID="6a43b30a-ac61-44d7-96b2-9925d08625df" containerID="f60cf4a563c31cf70cb0503f7eca8d8a0b52b964ef9c9c38851842383477bd13" exitCode=0 Nov 02 15:47:52.451824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:52.448470 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" event={"ID":"6a43b30a-ac61-44d7-96b2-9925d08625df","Type":"ContainerDied","Data":"f60cf4a563c31cf70cb0503f7eca8d8a0b52b964ef9c9c38851842383477bd13"} Nov 02 15:47:53.838544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:53.838515 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:54.023500 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.023463 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\") pod \"6a43b30a-ac61-44d7-96b2-9925d08625df\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " Nov 02 15:47:54.023670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.023520 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\") pod \"6a43b30a-ac61-44d7-96b2-9925d08625df\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " Nov 02 15:47:54.023670 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.023615 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\" (UniqueName: \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\") pod \"6a43b30a-ac61-44d7-96b2-9925d08625df\" (UID: \"6a43b30a-ac61-44d7-96b2-9925d08625df\") " Nov 02 15:47:54.023982 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.023948 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca" pod "6a43b30a-ac61-44d7-96b2-9925d08625df" (UID: "6a43b30a-ac61-44d7-96b2-9925d08625df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:54.025551 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.025529 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca" pod "6a43b30a-ac61-44d7-96b2-9925d08625df" (UID: "6a43b30a-ac61-44d7-96b2-9925d08625df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:54.025663 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.025628 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927" pod "6a43b30a-ac61-44d7-96b2-9925d08625df" (UID: "6a43b30a-ac61-44d7-96b2-9925d08625df"). InnerVolumeSpecName "kube-api-access-w7927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:54.124469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.124370 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7927\" (UniqueName: \"kubernetes.io/projected/6a43b30a-ac61-44d7-96b2-9925d08625df-kube-api-access-w7927\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.124469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.124416 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.124469 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.124430 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a43b30a-ac61-44d7-96b2-9925d08625df-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.475080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.475046 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" Nov 02 15:47:54.475251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.475079 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv" event={"ID":"6a43b30a-ac61-44d7-96b2-9925d08625df","Type":"ContainerDied","Data":"bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540"} Nov 02 15:47:54.475251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:54.475129 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc4ce44d54464e70cc5526a2d021ce18b167daf60ca2076e966d739ddaccf540" Nov 02 15:47:57.049019 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:57.048980 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m"] Nov 02 15:47:57.098447 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:57.098416 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rk77m"] Nov 02 15:47:57.237876 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:47:57.237845 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0fa9ad-3702-441d-a727-b15aa1e7412d" path="/var/lib/kubelet/pods/db0fa9ad-3702-441d-a727-b15aa1e7412d/volumes" Nov 02 15:48:00.761855 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:00.761789 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:04.660639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:04.660607 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:48:06.267096 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:48:06.267043 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:48:15.591781 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:15.591744 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:48:21.399123 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.399043 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv"] Nov 02 15:48:21.446997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.446963 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6lgcv"] Nov 02 15:48:21.523935 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.523903 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg"] Nov 02 15:48:21.524378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.524359 4469 cpu_manager.go:414] "containerMap: removing container" podUID="6a43b30a-ac61-44d7-96b2-9925d08625df" containerName="revoke-keys" Nov 02 15:48:21.524471 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.524380 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a43b30a-ac61-44d7-96b2-9925d08625df" containerName="revoke-keys" Nov 02 15:48:21.524531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.524523 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a43b30a-ac61-44d7-96b2-9925d08625df" containerName="revoke-keys" Nov 02 15:48:21.528012 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.527989 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.571617 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.571581 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.571767 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.571640 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.571866 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.571755 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8fh\" (UniqueName: \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.580056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.580032 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:48:21.580246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.580223 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:48:21.580363 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.580353 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:48:21.583368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.583350 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:48:21.586920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.586887 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg"] Nov 02 15:48:21.673086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.673003 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.673086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.673052 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.673086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.673083 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8fh\" (UniqueName: \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.673811 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.673776 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.675367 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.675342 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.734200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.734160 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8fh\" (UniqueName: \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\") pod \"maas-api-revoke-sub-3d4156f1-hzgcg\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:21.840813 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:21.840763 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:22.219835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:22.219809 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg"] Nov 02 15:48:22.221943 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:48:22.221913 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a08c9d_ad70_46f8_ba0e_238125f00978.slice/crio-d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391 WatchSource:0}: Error finding container d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391: Status 404 returned error can't find the container with id d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391 Nov 02 15:48:22.667719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:22.667684 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" event={"ID":"70a08c9d-ad70-46f8-ba0e-238125f00978","Type":"ContainerStarted","Data":"4408c8369f56aa45dc18714dceefa90cf0d7d9418b58ff18372f3a4b14278ccd"} Nov 02 15:48:22.668135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:22.667742 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" event={"ID":"70a08c9d-ad70-46f8-ba0e-238125f00978","Type":"ContainerStarted","Data":"d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391"} Nov 02 15:48:22.784300 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:22.784209 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" podStartSLOduration=1.7841923400000002 podStartE2EDuration="1.78419234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:22.782315732 +0000 UTC m=+3105.665918253" watchObservedRunningTime="2026-11-02 15:48:22.78419234 +0000 UTC m=+3105.667794860" Nov 02 15:48:23.237294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:23.237261 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a43b30a-ac61-44d7-96b2-9925d08625df" path="/var/lib/kubelet/pods/6a43b30a-ac61-44d7-96b2-9925d08625df/volumes" Nov 02 15:48:23.675628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:23.675592 4469 generic.go:356] "Generic (PLEG): container finished" podID="70a08c9d-ad70-46f8-ba0e-238125f00978" containerID="4408c8369f56aa45dc18714dceefa90cf0d7d9418b58ff18372f3a4b14278ccd" exitCode=0 Nov 02 15:48:23.676041 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:23.675671 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" event={"ID":"70a08c9d-ad70-46f8-ba0e-238125f00978","Type":"ContainerDied","Data":"4408c8369f56aa45dc18714dceefa90cf0d7d9418b58ff18372f3a4b14278ccd"} Nov 02 15:48:24.676711 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.676680 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:24.815678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.815657 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:24.902406 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.902366 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\" (UniqueName: \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\") pod \"70a08c9d-ad70-46f8-ba0e-238125f00978\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " Nov 02 15:48:24.902559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.902422 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\") pod \"70a08c9d-ad70-46f8-ba0e-238125f00978\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " Nov 02 15:48:24.902559 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.902508 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\") pod \"70a08c9d-ad70-46f8-ba0e-238125f00978\" (UID: \"70a08c9d-ad70-46f8-ba0e-238125f00978\") " Nov 02 15:48:24.902846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.902821 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca" pod "70a08c9d-ad70-46f8-ba0e-238125f00978" (UID: "70a08c9d-ad70-46f8-ba0e-238125f00978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:24.904477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.904457 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca" pod "70a08c9d-ad70-46f8-ba0e-238125f00978" (UID: "70a08c9d-ad70-46f8-ba0e-238125f00978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:24.904602 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:24.904588 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh" pod "70a08c9d-ad70-46f8-ba0e-238125f00978" (UID: "70a08c9d-ad70-46f8-ba0e-238125f00978"). InnerVolumeSpecName "kube-api-access-vx8fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:25.003700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.003673 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8fh\" (UniqueName: \"kubernetes.io/projected/70a08c9d-ad70-46f8-ba0e-238125f00978-kube-api-access-vx8fh\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.003700 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.003699 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.003904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.003709 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a08c9d-ad70-46f8-ba0e-238125f00978-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.688924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.688888 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" Nov 02 15:48:25.688924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.688908 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg" event={"ID":"70a08c9d-ad70-46f8-ba0e-238125f00978","Type":"ContainerDied","Data":"d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391"} Nov 02 15:48:25.689351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:25.688974 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d803a9e0b830cd40ffddffc63584ce31f7f7d2a40146ec67e31f206951e5c391" Nov 02 15:48:36.185677 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:48:36.185640 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:48:36.259924 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:36.259893 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:48:46.689227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:46.689190 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:48:51.142124 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:48:51.142076 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:01.115331 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:01.115295 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:04.049773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:04.049732 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2"] Nov 02 15:49:04.099347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:04.099311 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-krzz2"] Nov 02 15:49:05.237917 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:05.237886 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e184bd0c-d96b-49a2-b924-985bd9980ceb" path="/var/lib/kubelet/pods/e184bd0c-d96b-49a2-b924-985bd9980ceb/volumes" Nov 02 15:49:06.184087 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:49:06.184050 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:49:12.599675 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:12.599636 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:20.242139 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:20.242103 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:49:25.012143 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:25.012109 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:36.185923 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:49:36.185883 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:49:38.901778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.901746 4469 scope.go:122] "RemoveContainer" containerID="7de8566fec9bf7503a294cf3b8a2b54f33d847435699853964281abbbf486180" Nov 02 15:49:38.910221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.910196 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x"] Nov 02 15:49:38.910592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.910580 4469 cpu_manager.go:414] "containerMap: removing container" podUID="70a08c9d-ad70-46f8-ba0e-238125f00978" containerName="revoke-keys" Nov 02 15:49:38.910640 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.910593 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a08c9d-ad70-46f8-ba0e-238125f00978" containerName="revoke-keys" Nov 02 15:49:38.910687 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.910677 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a08c9d-ad70-46f8-ba0e-238125f00978" containerName="revoke-keys" Nov 02 15:49:38.915010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.914993 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:38.954042 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.954017 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:38.954146 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.954068 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:38.954191 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.954168 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpbx\" (UniqueName: \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:38.967538 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.967519 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:49:38.967618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.967602 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:49:38.967824 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:38.967790 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:49:39.019464 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.019441 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x"] Nov 02 15:49:39.053965 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.053939 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb"] Nov 02 15:49:39.055299 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.055265 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.055417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.055333 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.055417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.055394 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpbx\" (UniqueName: \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.056030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.056010 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.056181 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.056165 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tfpwb"] Nov 02 15:49:39.057625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.057604 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.068046 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.068027 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:49:39.117301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.117268 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpbx\" (UniqueName: \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\") pod \"maas-api-revoke-sub-e2fc9c5b-ppm8x\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.228658 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.228631 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:39.237120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.237094 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd77af1-9903-4d33-89e5-b7ad9590ff61" path="/var/lib/kubelet/pods/1cd77af1-9903-4d33-89e5-b7ad9590ff61/volumes" Nov 02 15:49:39.599296 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:39.599272 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x"] Nov 02 15:49:39.601147 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:49:39.601124 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bd2212_aebd_40e3_900c_e006248006f1.slice/crio-c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa WatchSource:0}: Error finding container c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa: Status 404 returned error can't find the container with id c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa Nov 02 15:49:40.093707 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:40.093665 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" event={"ID":"64bd2212-aebd-40e3-900c-e006248006f1","Type":"ContainerStarted","Data":"65e3810feb7417a65aebf51744913ea48b22cd18c7a49de78e88f55202e995a2"} Nov 02 15:49:40.094109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:40.093756 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" event={"ID":"64bd2212-aebd-40e3-900c-e006248006f1","Type":"ContainerStarted","Data":"c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa"} Nov 02 15:49:40.211980 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:40.211929 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" podStartSLOduration=2.211914349 podStartE2EDuration="2.211914349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:40.210135321 +0000 UTC m=+3183.093737839" watchObservedRunningTime="2026-11-02 15:49:40.211914349 +0000 UTC m=+3183.095516865" Nov 02 15:49:41.099973 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:41.099942 4469 generic.go:356] "Generic (PLEG): container finished" podID="64bd2212-aebd-40e3-900c-e006248006f1" containerID="65e3810feb7417a65aebf51744913ea48b22cd18c7a49de78e88f55202e995a2" exitCode=0 Nov 02 15:49:41.100394 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:41.100033 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" event={"ID":"64bd2212-aebd-40e3-900c-e006248006f1","Type":"ContainerDied","Data":"65e3810feb7417a65aebf51744913ea48b22cd18c7a49de78e88f55202e995a2"} Nov 02 15:49:42.241460 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.241433 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:42.287860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.287821 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\") pod \"64bd2212-aebd-40e3-900c-e006248006f1\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " Nov 02 15:49:42.287992 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.287953 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\") pod \"64bd2212-aebd-40e3-900c-e006248006f1\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " Nov 02 15:49:42.288032 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.287995 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\" (UniqueName: \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\") pod \"64bd2212-aebd-40e3-900c-e006248006f1\" (UID: \"64bd2212-aebd-40e3-900c-e006248006f1\") " Nov 02 15:49:42.288287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.288264 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca" pod "64bd2212-aebd-40e3-900c-e006248006f1" (UID: "64bd2212-aebd-40e3-900c-e006248006f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:42.289910 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.289887 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca" pod "64bd2212-aebd-40e3-900c-e006248006f1" (UID: "64bd2212-aebd-40e3-900c-e006248006f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:42.290055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.290033 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx" pod "64bd2212-aebd-40e3-900c-e006248006f1" (UID: "64bd2212-aebd-40e3-900c-e006248006f1"). InnerVolumeSpecName "kube-api-access-ffpbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:42.389479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.389411 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:42.389479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.389439 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bd2212-aebd-40e3-900c-e006248006f1-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:42.389479 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:42.389450 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpbx\" (UniqueName: \"kubernetes.io/projected/64bd2212-aebd-40e3-900c-e006248006f1-kube-api-access-ffpbx\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:43.111280 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:43.111250 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" Nov 02 15:49:43.111466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:43.111296 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x" event={"ID":"64bd2212-aebd-40e3-900c-e006248006f1","Type":"ContainerDied","Data":"c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa"} Nov 02 15:49:43.111466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:43.111351 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c4d78e8b8aefdd99980760ecf4e0395825b1bd0d3c0ecb44e3c10dc22c79fa" Nov 02 15:49:49.748720 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.748679 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb"] Nov 02 15:49:49.749319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.749298 4469 cpu_manager.go:414] "containerMap: removing container" podUID="64bd2212-aebd-40e3-900c-e006248006f1" containerName="revoke-keys" Nov 02 15:49:49.749419 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.749324 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bd2212-aebd-40e3-900c-e006248006f1" containerName="revoke-keys" Nov 02 15:49:49.749474 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.749430 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bd2212-aebd-40e3-900c-e006248006f1" containerName="revoke-keys" Nov 02 15:49:49.752769 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.752747 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.804882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.804838 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:49:49.805052 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.805011 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:49:49.805126 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.805075 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:49:49.805183 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.805139 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:49:49.856568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.856528 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb"] Nov 02 15:49:49.859756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.859733 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjsx\" (UniqueName: \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.859838 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.859784 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.859927 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.859907 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.961505 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.961472 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.961723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.961520 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.961723 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.961696 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjsx\" (UniqueName: \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.962273 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.962252 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:49.963959 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:49.963936 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:50.023400 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:50.023324 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjsx\" (UniqueName: \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\") pod \"maas-api-revoke-sub-5a29efb3-dzjwb\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:50.066115 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:50.066081 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:50.437860 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:50.437836 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb"] Nov 02 15:49:50.440152 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:49:50.440126 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4db765_d3dc_4500_a5d3_3839cf12820c.slice/crio-f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850 WatchSource:0}: Error finding container f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850: Status 404 returned error can't find the container with id f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850 Nov 02 15:49:51.050371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.050340 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n"] Nov 02 15:49:51.053135 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.053113 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8vv4n"] Nov 02 15:49:51.152257 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.152211 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" event={"ID":"2e4db765-d3dc-4500-a5d3-3839cf12820c","Type":"ContainerStarted","Data":"f9f6540015c8ba633d60fd894ccd300a82893249ede9116532c7dbcfc749bb6f"} Nov 02 15:49:51.152407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.152286 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" event={"ID":"2e4db765-d3dc-4500-a5d3-3839cf12820c","Type":"ContainerStarted","Data":"f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850"} Nov 02 15:49:51.240162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.240134 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020367fb-e237-4dbd-87de-050c1697ccc5" path="/var/lib/kubelet/pods/020367fb-e237-4dbd-87de-050c1697ccc5/volumes" Nov 02 15:49:51.268405 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.268347 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" podStartSLOduration=2.268331248 podStartE2EDuration="2.268331248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:51.267792527 +0000 UTC m=+3194.151395047" watchObservedRunningTime="2026-11-02 15:49:51.268331248 +0000 UTC m=+3194.151933770" Nov 02 15:49:51.492407 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.492372 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:51.861523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:51.861444 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:49:52.159786 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:52.159708 4469 generic.go:356] "Generic (PLEG): container finished" podID="2e4db765-d3dc-4500-a5d3-3839cf12820c" containerID="f9f6540015c8ba633d60fd894ccd300a82893249ede9116532c7dbcfc749bb6f" exitCode=0 Nov 02 15:49:52.160222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:52.159794 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" event={"ID":"2e4db765-d3dc-4500-a5d3-3839cf12820c","Type":"ContainerDied","Data":"f9f6540015c8ba633d60fd894ccd300a82893249ede9116532c7dbcfc749bb6f"} Nov 02 15:49:53.605678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.605334 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:49:53.720554 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.720519 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\") pod \"2e4db765-d3dc-4500-a5d3-3839cf12820c\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " Nov 02 15:49:53.720744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.720604 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\" (UniqueName: \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\") pod \"2e4db765-d3dc-4500-a5d3-3839cf12820c\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " Nov 02 15:49:53.720744 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.720664 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\") pod \"2e4db765-d3dc-4500-a5d3-3839cf12820c\" (UID: \"2e4db765-d3dc-4500-a5d3-3839cf12820c\") " Nov 02 15:49:53.720997 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.720969 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca" pod "2e4db765-d3dc-4500-a5d3-3839cf12820c" (UID: "2e4db765-d3dc-4500-a5d3-3839cf12820c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:53.722756 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.722728 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca" pod "2e4db765-d3dc-4500-a5d3-3839cf12820c" (UID: "2e4db765-d3dc-4500-a5d3-3839cf12820c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:53.722881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.722776 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx" pod "2e4db765-d3dc-4500-a5d3-3839cf12820c" (UID: "2e4db765-d3dc-4500-a5d3-3839cf12820c"). InnerVolumeSpecName "kube-api-access-xdjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:53.822513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.822422 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:53.822513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.822465 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjsx\" (UniqueName: \"kubernetes.io/projected/2e4db765-d3dc-4500-a5d3-3839cf12820c-kube-api-access-xdjsx\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:53.822513 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:53.822483 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4db765-d3dc-4500-a5d3-3839cf12820c-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:49:54.203931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:54.203226 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" event={"ID":"2e4db765-d3dc-4500-a5d3-3839cf12820c","Type":"ContainerDied","Data":"f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850"} Nov 02 15:49:54.203931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:54.203305 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3cdab8c098d77b7d9a99e4f37e7b5d8a0a50011e66618ded74df9ffc8291850" Nov 02 15:49:54.203931 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:49:54.203392 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb" Nov 02 15:50:02.557585 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:02.557546 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:50:06.186027 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:50:06.185992 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:50:06.468766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.468727 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg"] Nov 02 15:50:06.518090 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.518057 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hzgcg"] Nov 02 15:50:06.591716 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.591681 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x"] Nov 02 15:50:06.592196 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.592176 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2e4db765-d3dc-4500-a5d3-3839cf12820c" containerName="revoke-keys" Nov 02 15:50:06.592292 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.592198 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4db765-d3dc-4500-a5d3-3839cf12820c" containerName="revoke-keys" Nov 02 15:50:06.592350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.592291 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4db765-d3dc-4500-a5d3-3839cf12820c" containerName="revoke-keys" Nov 02 15:50:06.595914 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.595893 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.652522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.652480 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:50:06.652522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.652470 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:50:06.652522 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.652522 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:50:06.655136 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.655113 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:50:06.659069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.659041 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x"] Nov 02 15:50:06.694655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.694634 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.694791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.694700 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzbk\" (UniqueName: \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.694891 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.694872 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.795789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.795703 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzbk\" (UniqueName: \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.795789 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.795785 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.796057 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.795831 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.796458 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.796435 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.798168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.798145 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.857425 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.857394 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzbk\" (UniqueName: \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\") pod \"maas-api-revoke-sub-3d4156f1-dx78x\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:06.909389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:06.909356 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:07.237692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:07.237662 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a08c9d-ad70-46f8-ba0e-238125f00978" path="/var/lib/kubelet/pods/70a08c9d-ad70-46f8-ba0e-238125f00978/volumes" Nov 02 15:50:07.284725 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:07.284701 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x"] Nov 02 15:50:07.286936 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:50:07.286911 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d95f7b_501c_44bb_a3a9_186820b35ff8.slice/crio-04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8 WatchSource:0}: Error finding container 04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8: Status 404 returned error can't find the container with id 04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8 Nov 02 15:50:08.274403 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:08.274366 4469 generic.go:356] "Generic (PLEG): container finished" podID="b4d95f7b-501c-44bb-a3a9-186820b35ff8" containerID="5d0644422f121a2f57d2b345382cdfbd33eac469bee4561fa1ed1aede5b7627c" exitCode=0 Nov 02 15:50:08.274755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:08.274421 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" event={"ID":"b4d95f7b-501c-44bb-a3a9-186820b35ff8","Type":"ContainerDied","Data":"5d0644422f121a2f57d2b345382cdfbd33eac469bee4561fa1ed1aede5b7627c"} Nov 02 15:50:08.274755 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:08.274473 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" event={"ID":"b4d95f7b-501c-44bb-a3a9-186820b35ff8","Type":"ContainerStarted","Data":"04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8"} Nov 02 15:50:08.999519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:08.999483 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:09.408318 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.408294 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:09.521848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.521819 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\") pod \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " Nov 02 15:50:09.521989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.521894 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\" (UniqueName: \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\") pod \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " Nov 02 15:50:09.521989 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.521974 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\") pod \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\" (UID: \"b4d95f7b-501c-44bb-a3a9-186820b35ff8\") " Nov 02 15:50:09.522365 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.522327 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca" pod "b4d95f7b-501c-44bb-a3a9-186820b35ff8" (UID: "b4d95f7b-501c-44bb-a3a9-186820b35ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:50:09.524310 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.524281 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk" pod "b4d95f7b-501c-44bb-a3a9-186820b35ff8" (UID: "b4d95f7b-501c-44bb-a3a9-186820b35ff8"). InnerVolumeSpecName "kube-api-access-vzzbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:50:09.524428 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.524312 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca" pod "b4d95f7b-501c-44bb-a3a9-186820b35ff8" (UID: "b4d95f7b-501c-44bb-a3a9-186820b35ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:50:09.623381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.623349 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:50:09.623381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.623379 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d95f7b-501c-44bb-a3a9-186820b35ff8-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:50:09.623568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:09.623389 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzbk\" (UniqueName: \"kubernetes.io/projected/b4d95f7b-501c-44bb-a3a9-186820b35ff8-kube-api-access-vzzbk\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:50:10.292417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:10.292377 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" event={"ID":"b4d95f7b-501c-44bb-a3a9-186820b35ff8","Type":"ContainerDied","Data":"04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8"} Nov 02 15:50:10.292417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:10.292411 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x" Nov 02 15:50:10.292679 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:10.292452 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04428b91da1d7ccf8137c1fad877b8a9b2b7b957e964f45489172cbccdf360b8" Nov 02 15:50:21.345307 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:21.345266 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:50:25.359446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:25.359413 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:27.049778 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:27.049738 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz"] Nov 02 15:50:27.099264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:27.099234 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6stqz"] Nov 02 15:50:27.237330 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:27.237289 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b895d5-f307-45cd-aca4-1f9b6c0038fe" path="/var/lib/kubelet/pods/b5b895d5-f307-45cd-aca4-1f9b6c0038fe/volumes" Nov 02 15:50:31.845856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:31.845823 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:50:36.185449 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:50:36.185406 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:50:37.049816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:37.049772 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f"] Nov 02 15:50:37.099208 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:37.099175 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z4t4f"] Nov 02 15:50:37.246475 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:37.246442 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2feb2494-c7b8-4f7e-aa13-89efa6d80de7" path="/var/lib/kubelet/pods/2feb2494-c7b8-4f7e-aa13-89efa6d80de7/volumes" Nov 02 15:50:38.923281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:38.923249 4469 scope.go:122] "RemoveContainer" containerID="fd4b2d11b9d6e4a7ec14b38a2b68d5726e01f69375429c251df053de516854d7" Nov 02 15:50:38.932030 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:38.932009 4469 scope.go:122] "RemoveContainer" containerID="d8f51315c093a608dfd75f251816bee71d4dce31f9054862b2937c5ba61532f3" Nov 02 15:50:41.237555 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:41.237518 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:41.450382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:41.450345 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:44.197439 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:44.197405 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:44.414816 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:44.414770 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:44.641166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:44.641131 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:47.438568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:47.438533 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:47.637297 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:47.637246 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:50.407984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:50.407944 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:50.610782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:50.610753 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:51.830785 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:51.830747 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:55.351996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:55.351952 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:58.358631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:50:58.358590 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:51:00.312654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:00.312605 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:05.167188 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:05.167151 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:06.184831 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:51:06.184786 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:51:06.469358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:06.469325 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:09.451637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:09.451602 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:10.037497 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:10.037466 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:10.265409 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:10.265373 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:10.488631 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:10.488586 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:13.048676 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:13.048646 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n"] Nov 02 15:51:13.097182 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:13.097155 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p9z8n"] Nov 02 15:51:13.238432 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:13.238398 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa37ca25-8172-4419-8ca5-835a9b49b87b" path="/var/lib/kubelet/pods/fa37ca25-8172-4419-8ca5-835a9b49b87b/volumes" Nov 02 15:51:13.875199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:13.875143 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:15.363743 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:15.363707 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:18.393316 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:18.393282 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:51:20.255529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:20.255412 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:25.053301 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:25.053256 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:25.706179 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:25.706149 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:29.862606 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:29.862570 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:34.632255 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:34.632208 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:36.186157 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:51:36.186115 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:51:38.958322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:38.958287 4469 scope.go:122] "RemoveContainer" containerID="6269fd4b2d169131ebcccbeb733cb612459e8df2b69f4dda790606faa829621d" Nov 02 15:51:38.966722 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:38.966698 4469 scope.go:122] "RemoveContainer" containerID="9ba0f0211124ff5e8316568eee6c6390608e7587d55dd280ccab12ca00355811" Nov 02 15:51:38.974244 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:38.974216 4469 scope.go:122] "RemoveContainer" containerID="137f0d117c9438d4559209f1c6246f4a0178a278d938cba6067490f14dd5c819" Nov 02 15:51:39.446993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:39.446951 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:41.050765 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:41.050730 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z"] Nov 02 15:51:41.102114 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:41.102081 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8qt8z"] Nov 02 15:51:41.238062 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:41.238028 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fcdc4a-697d-4e91-815e-25c6ab054313" path="/var/lib/kubelet/pods/c0fcdc4a-697d-4e91-815e-25c6ab054313/volumes" Nov 02 15:51:44.261227 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:44.261180 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:49.059864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:49.059821 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:52.200369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:52.200333 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:53.858224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:53.858164 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:55.808546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:55.808512 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:51:58.682251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:51:58.682209 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:03.563270 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:03.563234 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:06.185783 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:52:06.185752 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:52:06.817467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:06.817431 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:08.361893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:08.361769 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:10.104614 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.104573 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp"] Nov 02 15:52:10.105193 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.105173 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b4d95f7b-501c-44bb-a3a9-186820b35ff8" containerName="revoke-keys" Nov 02 15:52:10.105276 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.105196 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d95f7b-501c-44bb-a3a9-186820b35ff8" containerName="revoke-keys" Nov 02 15:52:10.105366 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.105353 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d95f7b-501c-44bb-a3a9-186820b35ff8" containerName="revoke-keys" Nov 02 15:52:10.108487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.108466 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.160535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.160507 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:52:10.166190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.166159 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.166291 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.166243 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.166369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.166355 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmg2v\" (UniqueName: \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.210056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.210020 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:52:10.210185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.210085 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:52:10.210185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.210110 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:52:10.212134 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.212117 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp"] Nov 02 15:52:10.267222 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.267196 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmg2v\" (UniqueName: \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.267371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.267267 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.267371 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.267314 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.267897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.267881 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.269496 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.269475 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.328524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.328497 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmg2v\" (UniqueName: \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\") pod \"maas-api-revoke-sub-47cbd43d-crbzp\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.420954 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.420889 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:10.792285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.792261 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp"] Nov 02 15:52:10.794097 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:52:10.794069 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665f0efd_1b5f_4094_b911_a05367ec6f62.slice/crio-6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10 WatchSource:0}: Error finding container 6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10: Status 404 returned error can't find the container with id 6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10 Nov 02 15:52:10.959032 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:10.959002 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" event={"ID":"665f0efd-1b5f-4094-b911-a05367ec6f62","Type":"ContainerStarted","Data":"6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10"} Nov 02 15:52:11.965347 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:11.965309 4469 generic.go:356] "Generic (PLEG): container finished" podID="665f0efd-1b5f-4094-b911-a05367ec6f62" containerID="94679cdcf19cbfc3036c77bcd7825da183e36e50e146701547f5f56313e315ed" exitCode=0 Nov 02 15:52:11.965741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:11.965399 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" event={"ID":"665f0efd-1b5f-4094-b911-a05367ec6f62","Type":"ContainerDied","Data":"94679cdcf19cbfc3036c77bcd7825da183e36e50e146701547f5f56313e315ed"} Nov 02 15:52:12.068908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:12.068871 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx"] Nov 02 15:52:12.070680 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:12.070656 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tlbsx"] Nov 02 15:52:13.102171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.102149 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:13.153890 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.153775 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:13.199736 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.199704 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\" (UniqueName: \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\") pod \"665f0efd-1b5f-4094-b911-a05367ec6f62\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " Nov 02 15:52:13.200003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.199756 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\") pod \"665f0efd-1b5f-4094-b911-a05367ec6f62\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " Nov 02 15:52:13.200003 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.199864 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\") pod \"665f0efd-1b5f-4094-b911-a05367ec6f62\" (UID: \"665f0efd-1b5f-4094-b911-a05367ec6f62\") " Nov 02 15:52:13.200213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.200185 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca" pod "665f0efd-1b5f-4094-b911-a05367ec6f62" (UID: "665f0efd-1b5f-4094-b911-a05367ec6f62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:52:13.201944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.201916 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca" pod "665f0efd-1b5f-4094-b911-a05367ec6f62" (UID: "665f0efd-1b5f-4094-b911-a05367ec6f62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:52:13.202020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.201985 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v" pod "665f0efd-1b5f-4094-b911-a05367ec6f62" (UID: "665f0efd-1b5f-4094-b911-a05367ec6f62"). InnerVolumeSpecName "kube-api-access-zmg2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:52:13.239842 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.239754 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2027236e-5c94-4b25-8128-60e13e7bea98" path="/var/lib/kubelet/pods/2027236e-5c94-4b25-8128-60e13e7bea98/volumes" Nov 02 15:52:13.301678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.301645 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmg2v\" (UniqueName: \"kubernetes.io/projected/665f0efd-1b5f-4094-b911-a05367ec6f62-kube-api-access-zmg2v\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:13.301678 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.301677 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:13.301886 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.301687 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665f0efd-1b5f-4094-b911-a05367ec6f62-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:13.976699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.976666 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" event={"ID":"665f0efd-1b5f-4094-b911-a05367ec6f62","Type":"ContainerDied","Data":"6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10"} Nov 02 15:52:13.976907 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.976717 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a54add61f5e74c57c46cd6669250705fdd9e837e69b78d0faf547441e1c2b10" Nov 02 15:52:13.976907 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:13.976675 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp" Nov 02 15:52:17.969863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:17.969826 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:21.900628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:21.900588 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:22.392915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.392880 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5"] Nov 02 15:52:22.393404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.393385 4469 cpu_manager.go:414] "containerMap: removing container" podUID="665f0efd-1b5f-4094-b911-a05367ec6f62" containerName="revoke-keys" Nov 02 15:52:22.393526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.393405 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665f0efd-1b5f-4094-b911-a05367ec6f62" containerName="revoke-keys" Nov 02 15:52:22.393526 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.393498 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="665f0efd-1b5f-4094-b911-a05367ec6f62" containerName="revoke-keys" Nov 02 15:52:22.397048 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.397024 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.498084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498053 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fw8z\" (UniqueName: \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.498251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498107 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.498251 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498209 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.498717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498703 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:52:22.498717 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498709 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:52:22.498852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498703 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:52:22.498852 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.498703 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:52:22.500692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.500675 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5"] Nov 02 15:52:22.599636 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.599607 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fw8z\" (UniqueName: \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.599795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.599661 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.599795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.599694 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.600364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.600343 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.602140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.602111 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.669487 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.669432 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fw8z\" (UniqueName: \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.709812 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.709787 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:22.878837 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:22.878785 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:23.081978 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:23.081956 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5"] Nov 02 15:52:23.084106 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:52:23.084078 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5838655a_16a9_4c6d_94be_ba0314d2f052.slice/crio-a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb WatchSource:0}: Error finding container a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb: Status 404 returned error can't find the container with id a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb Nov 02 15:52:24.028616 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:24.028578 4469 generic.go:356] "Generic (PLEG): container finished" podID="5838655a-16a9-4c6d-94be-ba0314d2f052" containerID="8ff577b15b424412ff433c81b235fc4a4a4d58d7b871580cef238f962477bc6e" exitCode=0 Nov 02 15:52:24.028819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:24.028643 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" event={"ID":"5838655a-16a9-4c6d-94be-ba0314d2f052","Type":"ContainerDied","Data":"8ff577b15b424412ff433c81b235fc4a4a4d58d7b871580cef238f962477bc6e"} Nov 02 15:52:24.028819 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:24.028716 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" event={"ID":"5838655a-16a9-4c6d-94be-ba0314d2f052","Type":"ContainerStarted","Data":"a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb"} Nov 02 15:52:25.165304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.165277 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:52:25.172420 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.172401 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:25.329246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.329159 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\") pod \"5838655a-16a9-4c6d-94be-ba0314d2f052\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " Nov 02 15:52:25.329421 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.329370 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\" (UniqueName: \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\") pod \"5838655a-16a9-4c6d-94be-ba0314d2f052\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " Nov 02 15:52:25.329492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.329420 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\") pod \"5838655a-16a9-4c6d-94be-ba0314d2f052\" (UID: \"5838655a-16a9-4c6d-94be-ba0314d2f052\") " Nov 02 15:52:25.331519 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.331493 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca" pod "5838655a-16a9-4c6d-94be-ba0314d2f052" (UID: "5838655a-16a9-4c6d-94be-ba0314d2f052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:52:25.346198 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.332109 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z" pod "5838655a-16a9-4c6d-94be-ba0314d2f052" (UID: "5838655a-16a9-4c6d-94be-ba0314d2f052"). InnerVolumeSpecName "kube-api-access-5fw8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:52:25.346322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.333607 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca" pod "5838655a-16a9-4c6d-94be-ba0314d2f052" (UID: "5838655a-16a9-4c6d-94be-ba0314d2f052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:52:25.430632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.430598 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fw8z\" (UniqueName: \"kubernetes.io/projected/5838655a-16a9-4c6d-94be-ba0314d2f052-kube-api-access-5fw8z\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:25.430632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.430633 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:25.430906 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:25.430647 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5838655a-16a9-4c6d-94be-ba0314d2f052-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:26.039386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:26.039359 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" Nov 02 15:52:26.039581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:26.039348 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5" event={"ID":"5838655a-16a9-4c6d-94be-ba0314d2f052","Type":"ContainerDied","Data":"a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb"} Nov 02 15:52:26.039581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:26.039505 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31444acea55f6378c82e375f8d6f30a2a03f8f472c8808f61ad21db186690cb" Nov 02 15:52:26.110556 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:26.110525 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:26.925204 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:26.925169 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:28.068080 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:28.068033 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:32.860102 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:32.860063 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:34.323386 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:34.323335 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:34.526024 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:34.525992 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:36.184971 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:52:36.184939 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:52:37.079499 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:37.079463 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:38.621304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:38.621273 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:38.898879 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:38.898761 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:39.002771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:39.002740 4469 scope.go:122] "RemoveContainer" containerID="b96f317d0cbc4183b1c9e0882c260ed1d980ee44d283aa43f2f381087c27d386" Nov 02 15:52:39.011592 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:39.011573 4469 scope.go:122] "RemoveContainer" containerID="932ca2ec60b0006f381cef59bd7b7c93e30ec249c1ae4af16a15fc43075b811a" Nov 02 15:52:39.182063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:39.181969 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5"] Nov 02 15:52:39.236938 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:39.236910 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b72a1f-7x8h5"] Nov 02 15:52:39.711020 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:39.710987 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:41.239292 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:41.239259 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5838655a-16a9-4c6d-94be-ba0314d2f052" path="/var/lib/kubelet/pods/5838655a-16a9-4c6d-94be-ba0314d2f052/volumes" Nov 02 15:52:41.778467 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:41.778434 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:52:42.053339 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:42.053255 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf"] Nov 02 15:52:42.102929 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:42.102896 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lxfkf"] Nov 02 15:52:43.240706 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:43.240668 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e686eab-236c-41a0-8cb1-3172ea326a92" path="/var/lib/kubelet/pods/2e686eab-236c-41a0-8cb1-3172ea326a92/volumes" Nov 02 15:52:46.100994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:46.100891 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:46.308660 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:46.308629 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:46.523240 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:46.523206 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:48.145369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.145331 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x"] Nov 02 15:52:48.194758 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.194700 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dx78x"] Nov 02 15:52:48.276913 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.276878 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj"] Nov 02 15:52:48.277352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.277333 4469 cpu_manager.go:414] "containerMap: removing container" podUID="5838655a-16a9-4c6d-94be-ba0314d2f052" containerName="revoke-keys" Nov 02 15:52:48.277489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.277354 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5838655a-16a9-4c6d-94be-ba0314d2f052" containerName="revoke-keys" Nov 02 15:52:48.277589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.277507 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="5838655a-16a9-4c6d-94be-ba0314d2f052" containerName="revoke-keys" Nov 02 15:52:48.280817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.280782 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.333168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.333137 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:52:48.333168 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.333150 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:52:48.333360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.333250 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:52:48.333360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.333315 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:52:48.384288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.384256 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.384489 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.384300 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52m2\" (UniqueName: \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.384568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.384519 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.385774 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.385751 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj"] Nov 02 15:52:48.485996 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.485962 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.486187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.486035 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.486187 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.486064 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52m2\" (UniqueName: \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.486771 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.486751 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.488336 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.488316 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.546955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.546914 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52m2\" (UniqueName: \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\") pod \"maas-api-revoke-sub-3d4156f1-wzqlj\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.593845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.593822 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:48.976368 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:48.976337 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj"] Nov 02 15:52:48.978766 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:52:48.978736 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2672f2_7626_40a5_827b_a31f6766e280.slice/crio-e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19 WatchSource:0}: Error finding container e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19: Status 404 returned error can't find the container with id e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19 Nov 02 15:52:49.157285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:49.157257 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" event={"ID":"0d2672f2-7626-40a5-827b-a31f6766e280","Type":"ContainerStarted","Data":"e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19"} Nov 02 15:52:49.239544 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:49.239485 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d95f7b-501c-44bb-a3a9-186820b35ff8" path="/var/lib/kubelet/pods/b4d95f7b-501c-44bb-a3a9-186820b35ff8/volumes" Nov 02 15:52:50.163450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:50.163365 4469 generic.go:356] "Generic (PLEG): container finished" podID="0d2672f2-7626-40a5-827b-a31f6766e280" containerID="64f15d332c372d3fae1fcc60c8780f0111612d8b9df54a529f02ea9b03efb2cf" exitCode=0 Nov 02 15:52:50.163835 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:50.163450 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" event={"ID":"0d2672f2-7626-40a5-827b-a31f6766e280","Type":"ContainerDied","Data":"64f15d332c372d3fae1fcc60c8780f0111612d8b9df54a529f02ea9b03efb2cf"} Nov 02 15:52:51.300056 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.300034 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:51.419833 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.419743 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\") pod \"0d2672f2-7626-40a5-827b-a31f6766e280\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " Nov 02 15:52:51.419993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.419831 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\" (UniqueName: \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\") pod \"0d2672f2-7626-40a5-827b-a31f6766e280\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " Nov 02 15:52:51.420069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.420015 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\") pod \"0d2672f2-7626-40a5-827b-a31f6766e280\" (UID: \"0d2672f2-7626-40a5-827b-a31f6766e280\") " Nov 02 15:52:51.420190 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.420166 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca" pod "0d2672f2-7626-40a5-827b-a31f6766e280" (UID: "0d2672f2-7626-40a5-827b-a31f6766e280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:52:51.420364 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.420347 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:51.421998 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.421977 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca" pod "0d2672f2-7626-40a5-827b-a31f6766e280" (UID: "0d2672f2-7626-40a5-827b-a31f6766e280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:52:51.422120 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.422012 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2" pod "0d2672f2-7626-40a5-827b-a31f6766e280" (UID: "0d2672f2-7626-40a5-827b-a31f6766e280"). InnerVolumeSpecName "kube-api-access-p52m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:52:51.521482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.521455 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52m2\" (UniqueName: \"kubernetes.io/projected/0d2672f2-7626-40a5-827b-a31f6766e280-kube-api-access-p52m2\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:51.521482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:51.521481 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2672f2-7626-40a5-827b-a31f6766e280-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:52:52.175356 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:52.175321 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" Nov 02 15:52:52.175541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:52.175353 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj" event={"ID":"0d2672f2-7626-40a5-827b-a31f6766e280","Type":"ContainerDied","Data":"e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19"} Nov 02 15:52:52.175541 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:52.175408 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e525dd675d02e01580f4f04eed0e34e0968c37e8d25ff020b75480335087ec19" Nov 02 15:52:52.542446 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:52:52.542411 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:03.013881 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:03.013840 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:53:04.814450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:04.814410 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:05.030517 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:05.030465 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:06.186033 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:53:06.185995 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:53:07.179547 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:07.179501 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:11.117542 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:11.117485 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:13.672043 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:13.672003 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:53:22.911005 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:22.910958 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:53:22.911603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:22.911168 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" podUID="3b573780-88a8-4266-ae14-fdea1146f271" containerName="manager" containerID="cri-o://7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb" gracePeriod=10 Nov 02 15:53:23.167726 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.167670 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:53:23.291075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.291046 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\" (UniqueName: \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\") pod \"3b573780-88a8-4266-ae14-fdea1146f271\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " Nov 02 15:53:23.291221 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.291126 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\") pod \"3b573780-88a8-4266-ae14-fdea1146f271\" (UID: \"3b573780-88a8-4266-ae14-fdea1146f271\") " Nov 02 15:53:23.291466 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.291431 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume" pod "3b573780-88a8-4266-ae14-fdea1146f271" (UID: "3b573780-88a8-4266-ae14-fdea1146f271"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:53:23.293540 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.293513 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6" pod "3b573780-88a8-4266-ae14-fdea1146f271" (UID: "3b573780-88a8-4266-ae14-fdea1146f271"). InnerVolumeSpecName "kube-api-access-b6zs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:53:23.342211 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.342174 4469 generic.go:356] "Generic (PLEG): container finished" podID="3b573780-88a8-4266-ae14-fdea1146f271" containerID="7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb" exitCode=0 Nov 02 15:53:23.342359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.342222 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" Nov 02 15:53:23.342359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.342252 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" event={"ID":"3b573780-88a8-4266-ae14-fdea1146f271","Type":"ContainerDied","Data":"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb"} Nov 02 15:53:23.342359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.342310 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l" event={"ID":"3b573780-88a8-4266-ae14-fdea1146f271","Type":"ContainerDied","Data":"cc2755cc55c599ca700912acdd7c71d5c371ac7c52a1f65c3d06cc6e0c1b1890"} Nov 02 15:53:23.342359 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.342332 4469 scope.go:122] "RemoveContainer" containerID="7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb" Nov 02 15:53:23.355848 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.355829 4469 scope.go:122] "RemoveContainer" containerID="7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb" Nov 02 15:53:23.356135 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:53:23.356115 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb\": container with ID starting with 7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb not found: ID does not exist" containerID="7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb" Nov 02 15:53:23.356197 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.356145 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb"} err="failed to get container status \"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb\": rpc error: code = NotFound desc = could not find container \"7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb\": container with ID starting with 7c672a05707f73112afab3d1db3d79491c2b52a1ebba1ac48a09856d828c0ffb not found: ID does not exist" Nov 02 15:53:23.392395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.392370 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zs6\" (UniqueName: \"kubernetes.io/projected/3b573780-88a8-4266-ae14-fdea1146f271-kube-api-access-b6zs6\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:53:23.392395 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.392395 4469 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b573780-88a8-4266-ae14-fdea1146f271-extensions-socket-volume\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:53:23.518398 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.514342 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:53:23.523304 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:23.523277 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bnl8l"] Nov 02 15:53:25.237482 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:25.237446 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b573780-88a8-4266-ae14-fdea1146f271" path="/var/lib/kubelet/pods/3b573780-88a8-4266-ae14-fdea1146f271/volumes" Nov 02 15:53:26.438733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:26.438698 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:36.185760 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:53:36.185722 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:53:39.033932 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:39.033902 4469 scope.go:122] "RemoveContainer" containerID="043cd86743dd015b0c4258add54715dcd2014733e63ab76e081627c1c241bb4b" Nov 02 15:53:52.914863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:53:52.914823 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:06.185715 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:54:06.185681 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:54:07.555738 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:07.555706 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:26.623920 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.623847 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj"] Nov 02 15:54:26.624319 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624303 4469 cpu_manager.go:414] "containerMap: removing container" podUID="0d2672f2-7626-40a5-827b-a31f6766e280" containerName="revoke-keys" Nov 02 15:54:26.624362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624320 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2672f2-7626-40a5-827b-a31f6766e280" containerName="revoke-keys" Nov 02 15:54:26.624362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624345 4469 cpu_manager.go:414] "containerMap: removing container" podUID="3b573780-88a8-4266-ae14-fdea1146f271" containerName="manager" Nov 02 15:54:26.624362 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624350 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b573780-88a8-4266-ae14-fdea1146f271" containerName="manager" Nov 02 15:54:26.624465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624409 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b573780-88a8-4266-ae14-fdea1146f271" containerName="manager" Nov 02 15:54:26.624465 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.624421 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2672f2-7626-40a5-827b-a31f6766e280" containerName="revoke-keys" Nov 02 15:54:26.627590 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.627573 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.676278 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.676247 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj"] Nov 02 15:54:26.693249 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.693212 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9dd\" (UniqueName: \"kubernetes.io/projected/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-kube-api-access-9r9dd\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.693378 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.693339 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.723603 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.723579 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-22tph\"" Nov 02 15:54:26.794766 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.794735 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.794936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.794861 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9dd\" (UniqueName: \"kubernetes.io/projected/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-kube-api-access-9r9dd\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.795149 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.795128 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.813199 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.813168 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:26.868313 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.868287 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9dd\" (UniqueName: \"kubernetes.io/projected/f0c08a0e-075e-45fa-b789-cbcb33c3ffde-kube-api-access-9r9dd\") pod \"kuadrant-operator-controller-manager-678d87766f-qkfsj\" (UID: \"f0c08a0e-075e-45fa-b789-cbcb33c3ffde\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:26.939742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:26.939673 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:27.395452 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:27.395427 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj"] Nov 02 15:54:27.397903 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:54:27.397821 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c08a0e_075e_45fa_b789_cbcb33c3ffde.slice/crio-d9be8b665aacf44d48d57e788e64e14a8a8be38d7c70a98fa0288e12e3b9d30f WatchSource:0}: Error finding container d9be8b665aacf44d48d57e788e64e14a8a8be38d7c70a98fa0288e12e3b9d30f: Status 404 returned error can't find the container with id d9be8b665aacf44d48d57e788e64e14a8a8be38d7c70a98fa0288e12e3b9d30f Nov 02 15:54:27.664646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:27.664566 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" event={"ID":"f0c08a0e-075e-45fa-b789-cbcb33c3ffde","Type":"ContainerStarted","Data":"e96a9b7ba4dfd6706858823c8bc717815f1d80409b24dca11c99d56c4a435d80"} Nov 02 15:54:27.664646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:27.664622 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" event={"ID":"f0c08a0e-075e-45fa-b789-cbcb33c3ffde","Type":"ContainerStarted","Data":"d9be8b665aacf44d48d57e788e64e14a8a8be38d7c70a98fa0288e12e3b9d30f"} Nov 02 15:54:27.665117 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:27.664709 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:27.805953 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:27.805899 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" podStartSLOduration=1.8058846499999999 podStartE2EDuration="1.80588465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:27.799234168 +0000 UTC m=+3470.682836687" watchObservedRunningTime="2026-11-02 15:54:27.80588465 +0000 UTC m=+3470.689487169" Nov 02 15:54:36.184759 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:54:36.184721 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:54:38.672872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:38.672830 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qkfsj" Nov 02 15:54:39.058141 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:39.058106 4469 scope.go:122] "RemoveContainer" containerID="485bcb6a4d7efd5c8ca3834e456685f04c6ccfc35eb2c06dc65500d9520cda10" Nov 02 15:54:39.066714 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:39.066693 4469 scope.go:122] "RemoveContainer" containerID="4408c8369f56aa45dc18714dceefa90cf0d7d9418b58ff18372f3a4b14278ccd" Nov 02 15:54:39.074625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:39.074607 4469 scope.go:122] "RemoveContainer" containerID="f60cf4a563c31cf70cb0503f7eca8d8a0b52b964ef9c9c38851842383477bd13" Nov 02 15:54:43.055485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:43.055442 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x"] Nov 02 15:54:43.105002 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:43.104970 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ppm8x"] Nov 02 15:54:43.237109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:43.237081 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bd2212-aebd-40e3-900c-e006248006f1" path="/var/lib/kubelet/pods/64bd2212-aebd-40e3-900c-e006248006f1/volumes" Nov 02 15:54:46.781113 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:46.781076 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:54:51.225477 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.225437 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w"] Nov 02 15:54:51.229086 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.229070 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.285853 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.285795 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:54:51.332701 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.332661 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzjh\" (UniqueName: \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.332911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.332784 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.332911 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.332895 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.335306 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.335287 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:54:51.335536 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.335518 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:54:51.335625 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.335544 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:54:51.337546 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.337525 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w"] Nov 02 15:54:51.433634 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.433597 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzjh\" (UniqueName: \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.433864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.433669 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.433864 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.433702 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.434376 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.434352 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.436742 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.436713 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.508984 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.508914 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzjh\" (UniqueName: \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\") pod \"maas-api-revoke-sub-d576e68b-l2m6w\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.542885 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.542854 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:51.919846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:51.919818 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w"] Nov 02 15:54:51.921689 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:54:51.921660 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2410e0ed_bbd6_4d59_9f55_701fd8e59c4f.slice/crio-749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d WatchSource:0}: Error finding container 749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d: Status 404 returned error can't find the container with id 749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d Nov 02 15:54:52.810832 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:52.810776 4469 generic.go:356] "Generic (PLEG): container finished" podID="2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" containerID="5a547171b00a0b9d53f52d2ea08df60a89a60c3e2da7ac5f42864ae6f71f57cf" exitCode=0 Nov 02 15:54:52.811133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:52.810848 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" event={"ID":"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f","Type":"ContainerDied","Data":"5a547171b00a0b9d53f52d2ea08df60a89a60c3e2da7ac5f42864ae6f71f57cf"} Nov 02 15:54:52.811133 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:52.810901 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" event={"ID":"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f","Type":"ContainerStarted","Data":"749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d"} Nov 02 15:54:53.344370 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:53.344337 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:53.956699 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:53.956672 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:54.060063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.060027 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\") pod \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " Nov 02 15:54:54.060281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.060101 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\") pod \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " Nov 02 15:54:54.060281 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.060202 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\" (UniqueName: \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\") pod \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\" (UID: \"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f\") " Nov 02 15:54:54.060578 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.060470 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca" pod "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" (UID: "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:54:54.061128 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.061103 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb"] Nov 02 15:54:54.062375 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.062350 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca" pod "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" (UID: "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:54:54.062782 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.062764 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh" pod "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" (UID: "2410e0ed-bbd6-4d59-9f55-701fd8e59c4f"). InnerVolumeSpecName "kube-api-access-hvzjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:54:54.063417 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.063398 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dzjwb"] Nov 02 15:54:54.161007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.160970 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.161007 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.161002 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.161200 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.161017 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzjh\" (UniqueName: \"kubernetes.io/projected/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f-kube-api-access-hvzjh\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.821719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.821679 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" event={"ID":"2410e0ed-bbd6-4d59-9f55-701fd8e59c4f","Type":"ContainerDied","Data":"749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d"} Nov 02 15:54:54.821719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.821716 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w" Nov 02 15:54:54.821963 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:54.821767 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="749ed4061cea0431d5d0e9afd8e92fddd2f652d25c4df32d5db36e2b1e2fca7d" Nov 02 15:54:55.237818 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:54:55.237766 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4db765-d3dc-4500-a5d3-3839cf12820c" path="/var/lib/kubelet/pods/2e4db765-d3dc-4500-a5d3-3839cf12820c/volumes" Nov 02 15:55:04.136523 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:04.136483 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:55:06.186545 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:55:06.186501 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:55:07.993904 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:07.993864 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:13.864791 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:13.864754 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:55:20.693999 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.693960 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x"] Nov 02 15:55:20.694402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.694378 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" containerName="revoke-keys" Nov 02 15:55:20.694402 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.694390 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" containerName="revoke-keys" Nov 02 15:55:20.694494 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.694461 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" containerName="revoke-keys" Nov 02 15:55:20.697558 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.697539 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.751109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.751082 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:55:20.765566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.765537 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zct\" (UniqueName: \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.765728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.765646 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.765728 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.765701 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.800450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.800424 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:55:20.800646 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.800633 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:55:20.800691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.800647 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:55:20.802645 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.802626 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x"] Nov 02 15:55:20.866712 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.866684 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.866877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.866841 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zct\" (UniqueName: \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.866936 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.866888 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.867369 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.867344 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.869084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.869060 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:20.928724 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:20.928695 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zct\" (UniqueName: \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\") pod \"maas-api-revoke-sub-d2bc93bc-2fv4x\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:21.010535 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:21.010511 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:21.383788 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:21.383762 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x"] Nov 02 15:55:21.386163 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:55:21.386137 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444bf30f_2c11_427c_bd7c_f20b760e2537.slice/crio-86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48 WatchSource:0}: Error finding container 86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48: Status 404 returned error can't find the container with id 86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48 Nov 02 15:55:21.962882 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:21.962845 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" event={"ID":"444bf30f-2c11-427c-bd7c-f20b760e2537","Type":"ContainerStarted","Data":"f384418b54ba0e4fe54c2c25ede9d78a4c1ba4bc285fb898e46bb053d48fa35f"} Nov 02 15:55:21.963272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:21.962909 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" event={"ID":"444bf30f-2c11-427c-bd7c-f20b760e2537","Type":"ContainerStarted","Data":"86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48"} Nov 02 15:55:22.969285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:22.969250 4469 generic.go:356] "Generic (PLEG): container finished" podID="444bf30f-2c11-427c-bd7c-f20b760e2537" containerID="f384418b54ba0e4fe54c2c25ede9d78a4c1ba4bc285fb898e46bb053d48fa35f" exitCode=0 Nov 02 15:55:22.969683 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:22.969336 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" event={"ID":"444bf30f-2c11-427c-bd7c-f20b760e2537","Type":"ContainerDied","Data":"f384418b54ba0e4fe54c2c25ede9d78a4c1ba4bc285fb898e46bb053d48fa35f"} Nov 02 15:55:24.107579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.107552 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:24.203084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.203054 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\") pod \"444bf30f-2c11-427c-bd7c-f20b760e2537\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " Nov 02 15:55:24.203238 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.203107 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\" (UniqueName: \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\") pod \"444bf30f-2c11-427c-bd7c-f20b760e2537\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " Nov 02 15:55:24.203284 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.203261 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\") pod \"444bf30f-2c11-427c-bd7c-f20b760e2537\" (UID: \"444bf30f-2c11-427c-bd7c-f20b760e2537\") " Nov 02 15:55:24.203423 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.203395 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca" pod "444bf30f-2c11-427c-bd7c-f20b760e2537" (UID: "444bf30f-2c11-427c-bd7c-f20b760e2537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:55:24.203654 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.203637 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:55:24.205236 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.205212 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct" pod "444bf30f-2c11-427c-bd7c-f20b760e2537" (UID: "444bf30f-2c11-427c-bd7c-f20b760e2537"). InnerVolumeSpecName "kube-api-access-w8zct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:55:24.205340 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.205268 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca" pod "444bf30f-2c11-427c-bd7c-f20b760e2537" (UID: "444bf30f-2c11-427c-bd7c-f20b760e2537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:55:24.305055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.304972 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/444bf30f-2c11-427c-bd7c-f20b760e2537-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:55:24.305055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.305005 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zct\" (UniqueName: \"kubernetes.io/projected/444bf30f-2c11-427c-bd7c-f20b760e2537-kube-api-access-w8zct\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:55:24.980692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.980661 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" Nov 02 15:55:24.980692 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.980673 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x" event={"ID":"444bf30f-2c11-427c-bd7c-f20b760e2537","Type":"ContainerDied","Data":"86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48"} Nov 02 15:55:24.980929 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:24.980720 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86994d6fd78490762fae067459890db33a6799d9d195fb04d56c1a76343b7e48" Nov 02 15:55:25.144861 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:25.144819 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:25.352335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:25.352257 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:27.137132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:27.137097 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:33.154229 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:33.154191 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:33.564581 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:33.564551 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:55:33.840956 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:33.840874 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:34.093741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:34.093632 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:36.184767 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:55:36.184729 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:55:37.793355 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:37.793315 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:38.144404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:38.144323 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:41.008152 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:41.008109 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:41.211843 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:41.211792 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:44.191618 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:44.191564 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:44.454106 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:44.454022 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:44.658795 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:44.658763 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:49.151894 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:49.151863 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:52.063962 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:52.063924 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:55:53.606566 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:53.606530 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:54.102988 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:54.102946 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:58.949077 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:55:58.947595 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:02.730381 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:02.730345 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:03.341775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:03.341740 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:03.556322 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:03.556280 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:04.097589 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:04.097554 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:06.045360 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:06.045329 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:06.185473 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:56:06.185439 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:56:08.252241 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:08.252209 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:08.983166 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:08.983132 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:10.578408 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:10.578372 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:56:13.854995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:13.854962 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:18.655450 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:18.655418 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:23.451185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:23.451150 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:27.481033 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:27.481000 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:28.236979 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:28.236940 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:33.089173 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:33.089128 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:36.184954 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:56:36.184917 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:56:37.162397 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:37.162363 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:56:37.864579 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:37.864544 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:39.112140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:39.112107 4469 scope.go:122] "RemoveContainer" containerID="5d0644422f121a2f57d2b345382cdfbd33eac469bee4561fa1ed1aede5b7627c" Nov 02 15:56:39.121643 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:39.121619 4469 scope.go:122] "RemoveContainer" containerID="f9f6540015c8ba633d60fd894ccd300a82893249ede9116532c7dbcfc749bb6f" Nov 02 15:56:39.129418 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:39.129397 4469 scope.go:122] "RemoveContainer" containerID="65e3810feb7417a65aebf51744913ea48b22cd18c7a49de78e88f55202e995a2" Nov 02 15:56:42.739389 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:42.739347 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:47.562035 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:47.561994 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:52.361264 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:52.361222 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:53.963516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:53.963476 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:55.679995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:55.679961 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:57.771093 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:56:57.771053 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:01.962109 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:01.962074 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:02.222827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.222719 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt"] Nov 02 15:57:02.223132 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.223119 4469 cpu_manager.go:414] "containerMap: removing container" podUID="444bf30f-2c11-427c-bd7c-f20b760e2537" containerName="revoke-keys" Nov 02 15:57:02.223176 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.223133 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444bf30f-2c11-427c-bd7c-f20b760e2537" containerName="revoke-keys" Nov 02 15:57:02.223224 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.223215 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="444bf30f-2c11-427c-bd7c-f20b760e2537" containerName="revoke-keys" Nov 02 15:57:02.226325 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.226305 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.278730 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.278694 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:57:02.328282 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.328252 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:57:02.328495 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.328485 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:57:02.328642 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.328621 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:57:02.330575 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.330549 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt"] Nov 02 15:57:02.408433 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.408396 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.408610 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.408552 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.408721 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.408691 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bfq\" (UniqueName: \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.510037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.509948 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.510037 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.510002 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bfq\" (UniqueName: \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.510287 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.510046 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.510733 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.510713 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.512315 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.512291 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.571348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.571320 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bfq\" (UniqueName: \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\") pod \"maas-api-revoke-sub-52fab16a-dxbkt\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:02.839095 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:02.839001 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:03.208343 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:03.208318 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt"] Nov 02 15:57:03.210539 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:57:03.210510 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacc1b2c_4ddc_46d0_a85e_d13d49b7fedf.slice/crio-c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096 WatchSource:0}: Error finding container c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096: Status 404 returned error can't find the container with id c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096 Nov 02 15:57:03.482695 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:03.482664 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" event={"ID":"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf","Type":"ContainerStarted","Data":"c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096"} Nov 02 15:57:04.490214 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:04.490175 4469 generic.go:356] "Generic (PLEG): container finished" podID="dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" containerID="0571d46da8ce2285974445dac60fb007e696430080f4110045825f12c50c2414" exitCode=0 Nov 02 15:57:04.490657 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:04.490265 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" event={"ID":"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf","Type":"ContainerDied","Data":"0571d46da8ce2285974445dac60fb007e696430080f4110045825f12c50c2414"} Nov 02 15:57:05.633074 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.633045 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:05.642050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.642030 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\" (UniqueName: \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\") pod \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " Nov 02 15:57:05.642157 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.642102 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\") pod \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " Nov 02 15:57:05.642231 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.642215 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\") pod \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\" (UID: \"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf\") " Nov 02 15:57:05.642548 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.642523 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca" pod "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" (UID: "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:57:05.643961 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.643940 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq" pod "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" (UID: "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf"). InnerVolumeSpecName "kube-api-access-v8bfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:57:05.644044 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.644004 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca" pod "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" (UID: "dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:57:05.743827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.743772 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:05.743827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.743814 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bfq\" (UniqueName: \"kubernetes.io/projected/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-kube-api-access-v8bfq\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:05.743827 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:05.743826 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:06.185192 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:57:06.185154 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:57:06.501637 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:06.501610 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" Nov 02 15:57:06.501856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:06.501617 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt" event={"ID":"dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf","Type":"ContainerDied","Data":"c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096"} Nov 02 15:57:06.501856 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:06.501727 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c823ad6420665209faa1be1bac9da9b8bb11742d074385832e6fb9878bed6096" Nov 02 15:57:06.526972 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:06.526948 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:06.752897 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:06.752440 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:08.616667 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:08.616635 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:11.635749 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:11.635716 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:13.462695 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.462649 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:13.885946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.885872 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft"] Nov 02 15:57:13.886305 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.886292 4469 cpu_manager.go:414] "containerMap: removing container" podUID="dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" containerName="revoke-keys" Nov 02 15:57:13.886348 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.886306 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" containerName="revoke-keys" Nov 02 15:57:13.886392 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.886383 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" containerName="revoke-keys" Nov 02 15:57:13.889350 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.889333 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:13.920530 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.920502 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vqq\" (UniqueName: \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:13.920639 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.920562 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:13.920681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.920639 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:13.941628 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.941598 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:57:13.991040 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.991002 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:57:13.991209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.991049 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:57:13.991209 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.991076 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:57:13.994437 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:13.994413 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft"] Nov 02 15:57:14.022010 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.021979 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.022162 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.022036 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.022220 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.022171 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vqq\" (UniqueName: \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.022775 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.022752 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.024351 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.024332 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.049948 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.049914 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp"] Nov 02 15:57:14.052912 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.052889 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbd43d-crbzp"] Nov 02 15:57:14.082823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.082775 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vqq\" (UniqueName: \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.201883 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.201777 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:14.576719 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:14.576692 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft"] Nov 02 15:57:14.578819 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:57:14.578768 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc65346_893c_451c_a740_cd43add04849.slice/crio-cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c WatchSource:0}: Error finding container cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c: Status 404 returned error can't find the container with id cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c Nov 02 15:57:15.237586 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:15.237550 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665f0efd-1b5f-4094-b911-a05367ec6f62" path="/var/lib/kubelet/pods/665f0efd-1b5f-4094-b911-a05367ec6f62/volumes" Nov 02 15:57:15.315462 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:15.306956 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:57:15.550870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:15.550762 4469 generic.go:356] "Generic (PLEG): container finished" podID="8dc65346-893c-451c-a740-cd43add04849" containerID="1cf5abe7f60f19eb461b96ab6415a4e1cf7e097731fff5fa169b7135f0a0d481" exitCode=0 Nov 02 15:57:15.550870 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:15.550821 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" event={"ID":"8dc65346-893c-451c-a740-cd43add04849","Type":"ContainerDied","Data":"1cf5abe7f60f19eb461b96ab6415a4e1cf7e097731fff5fa169b7135f0a0d481"} Nov 02 15:57:15.551069 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:15.550875 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" event={"ID":"8dc65346-893c-451c-a740-cd43add04849","Type":"ContainerStarted","Data":"cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c"} Nov 02 15:57:16.715966 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.715937 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:16.750946 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.750919 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\" (UniqueName: \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\") pod \"8dc65346-893c-451c-a740-cd43add04849\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " Nov 02 15:57:16.751084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.750966 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\") pod \"8dc65346-893c-451c-a740-cd43add04849\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " Nov 02 15:57:16.751084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.751059 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\") pod \"8dc65346-893c-451c-a740-cd43add04849\" (UID: \"8dc65346-893c-451c-a740-cd43add04849\") " Nov 02 15:57:16.751393 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.751371 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca" pod "8dc65346-893c-451c-a740-cd43add04849" (UID: "8dc65346-893c-451c-a740-cd43add04849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:57:16.753054 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.753033 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca" pod "8dc65346-893c-451c-a740-cd43add04849" (UID: "8dc65346-893c-451c-a740-cd43add04849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:57:16.753140 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.753107 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq" pod "8dc65346-893c-451c-a740-cd43add04849" (UID: "8dc65346-893c-451c-a740-cd43add04849"). InnerVolumeSpecName "kube-api-access-82vqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:57:16.804246 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.804212 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:16.852382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.852294 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vqq\" (UniqueName: \"kubernetes.io/projected/8dc65346-893c-451c-a740-cd43add04849-kube-api-access-82vqq\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:16.852382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.852323 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc65346-893c-451c-a740-cd43add04849-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:16.852382 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:16.852332 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc65346-893c-451c-a740-cd43add04849-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:57:17.561893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:17.561790 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" Nov 02 15:57:17.561893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:17.561823 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft" event={"ID":"8dc65346-893c-451c-a740-cd43add04849","Type":"ContainerDied","Data":"cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c"} Nov 02 15:57:17.561893 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:17.561870 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd327c4d60cfa83ecf05d2d8fcf1886ce6dbfc53b92c0a50c9c63b82c85a734c" Nov 02 15:57:18.482691 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:18.482650 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:21.856172 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:21.855935 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:25.787272 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:25.787127 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:25.985404 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:25.985377 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:26.710075 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:26.710042 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:27.833847 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:27.833788 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:30.870888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:30.870850 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:31.419290 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:31.419253 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:32.417653 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:32.417616 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:32.695655 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:32.695554 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:33.053863 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:33.053826 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft"] Nov 02 15:57:33.100999 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:33.100951 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a60cf3-fh4ft"] Nov 02 15:57:33.237796 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:33.237764 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc65346-893c-451c-a740-cd43add04849" path="/var/lib/kubelet/pods/8dc65346-893c-451c-a740-cd43add04849/volumes" Nov 02 15:57:33.495061 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:33.495016 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:35.464263 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:35.464223 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:57:36.184754 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:57:36.184716 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:57:39.795283 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:39.795242 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:39.999907 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:39.999864 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:40.208823 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:40.208685 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:47.097741 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:47.097703 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:51.483704 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:51.483663 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:57:52.072139 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:52.072096 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj"] Nov 02 15:57:52.121681 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:52.121647 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wzqlj"] Nov 02 15:57:53.261990 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:53.261953 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2672f2-7626-40a5-827b-a31f6766e280" path="/var/lib/kubelet/pods/0d2672f2-7626-40a5-827b-a31f6766e280/volumes" Nov 02 15:57:54.314908 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:54.314862 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:55.401896 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:55.401846 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:55.894529 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:57:55.894500 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:06.185712 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:58:06.185674 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:58:08.951971 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:08.951927 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:11.074569 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:11.074519 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:16.326185 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:16.326120 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:17.340995 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:17.340959 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:28.257050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:28.257013 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:29.738594 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:29.738543 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:34.279516 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:34.279481 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:58:36.216938 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:58:36.216901 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:58:39.167903 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:39.167872 4469 scope.go:122] "RemoveContainer" containerID="8ff577b15b424412ff433c81b235fc4a4a4d58d7b871580cef238f962477bc6e" Nov 02 15:58:39.176261 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:39.176236 4469 scope.go:122] "RemoveContainer" containerID="94679cdcf19cbfc3036c77bcd7825da183e36e50e146701547f5f56313e315ed" Nov 02 15:58:54.684564 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:58:54.684532 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:04.773303 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.773259 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m"] Nov 02 15:59:04.773947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.773870 4469 cpu_manager.go:414] "containerMap: removing container" podUID="8dc65346-893c-451c-a740-cd43add04849" containerName="revoke-keys" Nov 02 15:59:04.773947 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.773892 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc65346-893c-451c-a740-cd43add04849" containerName="revoke-keys" Nov 02 15:59:04.774084 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.774035 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc65346-893c-451c-a740-cd43add04849" containerName="revoke-keys" Nov 02 15:59:04.777219 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.777198 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.829486 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.829456 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:59:04.878949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.878909 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:59:04.878949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.878935 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:59:04.878949 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.878947 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:59:04.881058 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.881037 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m"] Nov 02 15:59:04.886352 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.886334 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9wq\" (UniqueName: \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.886445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.886374 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.886445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.886405 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.988181 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.988150 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9wq\" (UniqueName: \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.988345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.988202 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.988345 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.988237 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.988955 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.988932 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:04.990524 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:04.990507 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:05.048909 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:05.048841 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9wq\" (UniqueName: \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\") pod \"maas-api-revoke-sub-452b6d7b-8kc8m\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:05.089739 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:05.089712 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:05.464294 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:05.464266 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m"] Nov 02 15:59:05.467096 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:59:05.467062 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8402133_4e22_4864_ae24_8ae617e89425.slice/crio-d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4 WatchSource:0}: Error finding container d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4: Status 404 returned error can't find the container with id d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4 Nov 02 15:59:06.050784 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.050706 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:59:06.142445 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.142406 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" event={"ID":"a8402133-4e22-4864-ae24-8ae617e89425","Type":"ContainerStarted","Data":"db3ea7cac2337d107309810a5f3361400bb2397bcdf28079817536ec5fa60848"} Nov 02 15:59:06.142638 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.142472 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" event={"ID":"a8402133-4e22-4864-ae24-8ae617e89425","Type":"ContainerStarted","Data":"d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4"} Nov 02 15:59:06.185899 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:59:06.185855 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:59:06.259079 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.259031 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" podStartSLOduration=2.259016173 podStartE2EDuration="2.259016173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:59:06.258573564 +0000 UTC m=+3749.142176088" watchObservedRunningTime="2026-11-02 15:59:06.259016173 +0000 UTC m=+3749.142618693" Nov 02 15:59:06.697127 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.697087 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:06.899091 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:06.899042 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:07.150295 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:07.150252 4469 generic.go:356] "Generic (PLEG): container finished" podID="a8402133-4e22-4864-ae24-8ae617e89425" containerID="db3ea7cac2337d107309810a5f3361400bb2397bcdf28079817536ec5fa60848" exitCode=0 Nov 02 15:59:07.150845 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:07.150377 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" event={"ID":"a8402133-4e22-4864-ae24-8ae617e89425","Type":"ContainerDied","Data":"db3ea7cac2337d107309810a5f3361400bb2397bcdf28079817536ec5fa60848"} Nov 02 15:59:08.581055 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.581027 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:08.633267 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.633232 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\") pod \"a8402133-4e22-4864-ae24-8ae617e89425\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " Nov 02 15:59:08.633568 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.633529 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\" (UniqueName: \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\") pod \"a8402133-4e22-4864-ae24-8ae617e89425\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " Nov 02 15:59:08.633872 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.633851 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\") pod \"a8402133-4e22-4864-ae24-8ae617e89425\" (UID: \"a8402133-4e22-4864-ae24-8ae617e89425\") " Nov 02 15:59:08.640050 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.640025 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca" pod "a8402133-4e22-4864-ae24-8ae617e89425" (UID: "a8402133-4e22-4864-ae24-8ae617e89425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:59:08.660834 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.657307 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq" pod "a8402133-4e22-4864-ae24-8ae617e89425" (UID: "a8402133-4e22-4864-ae24-8ae617e89425"). InnerVolumeSpecName "kube-api-access-pv9wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:59:08.665358 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.664938 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca" pod "a8402133-4e22-4864-ae24-8ae617e89425" (UID: "a8402133-4e22-4864-ae24-8ae617e89425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:59:08.761875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.735460 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:08.761875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.735498 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8402133-4e22-4864-ae24-8ae617e89425-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:08.761875 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:08.735514 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv9wq\" (UniqueName: \"kubernetes.io/projected/a8402133-4e22-4864-ae24-8ae617e89425-kube-api-access-pv9wq\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:09.198877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:09.198066 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" event={"ID":"a8402133-4e22-4864-ae24-8ae617e89425","Type":"ContainerDied","Data":"d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4"} Nov 02 15:59:09.198877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:09.198151 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e1d47bc8872d84eefd798f731672b158efe3ac982d50c60bc974cb4f970cc4" Nov 02 15:59:09.198877 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:09.198245 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m" Nov 02 15:59:09.252780 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:59:09.252748 4469 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8402133_4e22_4864_ae24_8ae617e89425.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:59:09.314846 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:09.314727 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:17.827063 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:17.827027 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:21.473994 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:21.473957 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:59:22.609915 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:22.609867 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:26.912503 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.912471 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99"] Nov 02 15:59:26.912944 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.912926 4469 cpu_manager.go:414] "containerMap: removing container" podUID="a8402133-4e22-4864-ae24-8ae617e89425" containerName="revoke-keys" Nov 02 15:59:26.912993 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.912945 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8402133-4e22-4864-ae24-8ae617e89425" containerName="revoke-keys" Nov 02 15:59:26.913038 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.913031 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8402133-4e22-4864-ae24-8ae617e89425" containerName="revoke-keys" Nov 02 15:59:26.916363 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.916339 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:26.965632 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.965600 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:26.965773 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.965713 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:26.965850 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:26.965781 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqrr\" (UniqueName: \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.018285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.018253 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:59:27.018285 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.018266 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:59:27.018492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.018268 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 15:59:27.018492 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.018328 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:59:27.020317 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.020298 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99"] Nov 02 15:59:27.067130 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.067104 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.067288 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.067157 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.067335 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.067321 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqrr\" (UniqueName: \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.067763 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.067740 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.069531 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.069510 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.128239 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.128208 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqrr\" (UniqueName: \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\") pod \"maas-api-revoke-sub-3e31a506-sbb99\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.230309 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.230283 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:27.603021 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:27.602995 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99"] Nov 02 15:59:27.605102 ip-10-0-135-103 kubenswrapper[4469]: W1102 15:59:27.605072 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3438add3_359e_407a_95a5_6343f09fe688.slice/crio-20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235 WatchSource:0}: Error finding container 20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235: Status 404 returned error can't find the container with id 20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235 Nov 02 15:59:28.325601 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:28.325561 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" event={"ID":"3438add3-359e-407a-95a5-6343f09fe688","Type":"ContainerStarted","Data":"f185bbd18d75b737e95acb76f3a4334a3a1fc31f6dbf547a287f6201eb79d422"} Nov 02 15:59:28.326213 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:28.325617 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" event={"ID":"3438add3-359e-407a-95a5-6343f09fe688","Type":"ContainerStarted","Data":"20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235"} Nov 02 15:59:28.466478 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:28.466417 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" podStartSLOduration=2.466400926 podStartE2EDuration="2.466400926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:59:28.465993628 +0000 UTC m=+3771.349596147" watchObservedRunningTime="2026-11-02 15:59:28.466400926 +0000 UTC m=+3771.350003445" Nov 02 15:59:28.537036 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:28.537001 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:29.334902 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:29.334867 4469 generic.go:356] "Generic (PLEG): container finished" podID="3438add3-359e-407a-95a5-6343f09fe688" containerID="f185bbd18d75b737e95acb76f3a4334a3a1fc31f6dbf547a287f6201eb79d422" exitCode=0 Nov 02 15:59:29.335435 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:29.334990 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" event={"ID":"3438add3-359e-407a-95a5-6343f09fe688","Type":"ContainerDied","Data":"f185bbd18d75b737e95acb76f3a4334a3a1fc31f6dbf547a287f6201eb79d422"} Nov 02 15:59:30.810746 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.810592 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:30.919073 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.919043 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\" (UniqueName: \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\") pod \"3438add3-359e-407a-95a5-6343f09fe688\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " Nov 02 15:59:30.919279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.919172 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\") pod \"3438add3-359e-407a-95a5-6343f09fe688\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " Nov 02 15:59:30.919279 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.919199 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\") pod \"3438add3-359e-407a-95a5-6343f09fe688\" (UID: \"3438add3-359e-407a-95a5-6343f09fe688\") " Nov 02 15:59:30.919595 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.919570 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca" pod "3438add3-359e-407a-95a5-6343f09fe688" (UID: "3438add3-359e-407a-95a5-6343f09fe688"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:59:30.921320 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.921283 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr" pod "3438add3-359e-407a-95a5-6343f09fe688" (UID: "3438add3-359e-407a-95a5-6343f09fe688"). InnerVolumeSpecName "kube-api-access-4dqrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:59:30.921443 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:30.921428 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca" pod "3438add3-359e-407a-95a5-6343f09fe688" (UID: "3438add3-359e-407a-95a5-6343f09fe688"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:59:31.020485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.020449 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3438add3-359e-407a-95a5-6343f09fe688-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:31.020485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.020478 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqrr\" (UniqueName: \"kubernetes.io/projected/3438add3-359e-407a-95a5-6343f09fe688-kube-api-access-4dqrr\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:31.020485 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.020489 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3438add3-359e-407a-95a5-6343f09fe688-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 15:59:31.370552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.370467 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" event={"ID":"3438add3-359e-407a-95a5-6343f09fe688","Type":"ContainerDied","Data":"20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235"} Nov 02 15:59:31.370552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.370498 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99" Nov 02 15:59:31.370552 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.370530 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e79d2017680130b0a9d4148a9759ed3d73f3144fc93ee91c8770cc0e49d235" Nov 02 15:59:31.758171 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:31.752579 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:59:36.190861 ip-10-0-135-103 kubenswrapper[4469]: E1102 15:59:36.190819 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 15:59:39.198888 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:39.198857 4469 scope.go:122] "RemoveContainer" containerID="64f15d332c372d3fae1fcc60c8780f0111612d8b9df54a529f02ea9b03efb2cf" Nov 02 15:59:40.245817 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:40.245772 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:59:50.239549 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:50.239517 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 15:59:55.030008 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:55.029973 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:55.049506 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:55.049478 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w"] Nov 02 15:59:55.098760 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:55.098734 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l2m6w"] Nov 02 15:59:55.237929 ip-10-0-135-103 kubenswrapper[4469]: I1102 15:59:55.237901 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2410e0ed-bbd6-4d59-9f55-701fd8e59c4f" path="/var/lib/kubelet/pods/2410e0ed-bbd6-4d59-9f55-701fd8e59c4f/volumes" Nov 02 16:00:00.085352 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.085313 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n"] Nov 02 16:00:00.085759 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.085746 4469 cpu_manager.go:414] "containerMap: removing container" podUID="3438add3-359e-407a-95a5-6343f09fe688" containerName="revoke-keys" Nov 02 16:00:00.085832 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.085760 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3438add3-359e-407a-95a5-6343f09fe688" containerName="revoke-keys" Nov 02 16:00:00.085872 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.085849 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="3438add3-359e-407a-95a5-6343f09fe688" containerName="revoke-keys" Nov 02 16:00:00.089207 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.089190 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.133645 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.133608 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.133830 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.133784 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsc7\" (UniqueName: \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.133886 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.133862 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.142060 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.142042 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:00:00.191858 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.191817 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:00:00.191858 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.191844 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:00:00.192084 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.191907 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\"" Nov 02 16:00:00.193940 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.193919 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n"] Nov 02 16:00:00.235000 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.234965 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsc7\" (UniqueName: \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.235195 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.235017 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.235195 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.235088 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.235983 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.235945 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.237553 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.237527 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.296905 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.296867 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsc7\" (UniqueName: \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\") pod \"maas-api-revoke-sub-cd0cb807-zqx8n\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.401369 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.401294 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:00.771684 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:00.771659 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n"] Nov 02 16:00:00.773793 ip-10-0-135-103 kubenswrapper[4469]: W1102 16:00:00.773767 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a75b45e_f8be_49ca_9521_1c278c6486ae.slice/crio-cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79 WatchSource:0}: Error finding container cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79: Status 404 returned error can't find the container with id cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79 Nov 02 16:00:01.536749 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:01.536717 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" event={"ID":"2a75b45e-f8be-49ca-9521-1c278c6486ae","Type":"ContainerStarted","Data":"393d78a8e2f6e76a20e38850a80c6c0cfb78b664d570aa591d65f58aada69e5d"} Nov 02 16:00:01.537090 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:01.536773 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" event={"ID":"2a75b45e-f8be-49ca-9521-1c278c6486ae","Type":"ContainerStarted","Data":"cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79"} Nov 02 16:00:02.543976 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.543945 4469 generic.go:356] "Generic (PLEG): container finished" podID="2a75b45e-f8be-49ca-9521-1c278c6486ae" containerID="393d78a8e2f6e76a20e38850a80c6c0cfb78b664d570aa591d65f58aada69e5d" exitCode=0 Nov 02 16:00:02.544391 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.544029 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" event={"ID":"2a75b45e-f8be-49ca-9521-1c278c6486ae","Type":"ContainerDied","Data":"393d78a8e2f6e76a20e38850a80c6c0cfb78b664d570aa591d65f58aada69e5d"} Nov 02 16:00:02.676131 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.676109 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:02.761099 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.761063 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\" (UniqueName: \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\") pod \"2a75b45e-f8be-49ca-9521-1c278c6486ae\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " Nov 02 16:00:02.761257 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.761179 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\") pod \"2a75b45e-f8be-49ca-9521-1c278c6486ae\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " Nov 02 16:00:02.761321 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.761304 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\") pod \"2a75b45e-f8be-49ca-9521-1c278c6486ae\" (UID: \"2a75b45e-f8be-49ca-9521-1c278c6486ae\") " Nov 02 16:00:02.761558 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.761531 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca" pod "2a75b45e-f8be-49ca-9521-1c278c6486ae" (UID: "2a75b45e-f8be-49ca-9521-1c278c6486ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:00:02.761644 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.761627 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-openshift-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:02.763307 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.763279 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca" pod "2a75b45e-f8be-49ca-9521-1c278c6486ae" (UID: "2a75b45e-f8be-49ca-9521-1c278c6486ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:00:02.763430 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.763327 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7" pod "2a75b45e-f8be-49ca-9521-1c278c6486ae" (UID: "2a75b45e-f8be-49ca-9521-1c278c6486ae"). InnerVolumeSpecName "kube-api-access-kbsc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:00:02.863161 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.863079 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsc7\" (UniqueName: \"kubernetes.io/projected/2a75b45e-f8be-49ca-9521-1c278c6486ae-kube-api-access-kbsc7\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:02.863161 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:02.863110 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a75b45e-f8be-49ca-9521-1c278c6486ae-maas-api-xks-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:03.550334 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:03.550260 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" Nov 02 16:00:03.550691 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:03.550258 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n" event={"ID":"2a75b45e-f8be-49ca-9521-1c278c6486ae","Type":"ContainerDied","Data":"cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79"} Nov 02 16:00:03.550691 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:03.550382 4469 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd5f11aea4d3202887b1821a0c11388ca9b5aa9d32d4e5f4785f9c90411ec79" Nov 02 16:00:06.185264 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:00:06.185230 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:00:12.952282 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:12.952250 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:00:23.426060 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.425985 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:23.426419 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.426389 4469 cpu_manager.go:414] "containerMap: removing container" podUID="2a75b45e-f8be-49ca-9521-1c278c6486ae" containerName="revoke-keys" Nov 02 16:00:23.426419 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.426400 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a75b45e-f8be-49ca-9521-1c278c6486ae" containerName="revoke-keys" Nov 02 16:00:23.426510 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.426466 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a75b45e-f8be-49ca-9521-1c278c6486ae" containerName="revoke-keys" Nov 02 16:00:23.429731 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.429713 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.531598 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.531562 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 16:00:23.531831 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.531720 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:00:23.531831 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.531724 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vkjdq\"" Nov 02 16:00:23.531831 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.531771 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 16:00:23.531831 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.531793 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:00:23.534578 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.534556 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:23.557369 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.557343 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.557487 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.557472 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.557529 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.557500 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62xt\" (UniqueName: \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.557587 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.557567 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.658904 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.658872 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.658904 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.658908 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62xt\" (UniqueName: \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.659089 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.658928 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.659089 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.658952 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.659657 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.659637 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.661274 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.661251 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.661486 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.661466 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.719208 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.719177 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62xt\" (UniqueName: \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\") pod \"maas-api-88df9b747-w4c75\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:23.743101 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:23.743076 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:24.120551 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:24.120527 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:24.122407 ip-10-0-135-103 kubenswrapper[4469]: W1102 16:00:24.122382 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03c8af7_165b_4fe1_95d2_ed27c77d22f1.slice/crio-8138a93b86c6ef15a74f707e924d5961d636e7e911e1d4494804f56ca2dccead WatchSource:0}: Error finding container 8138a93b86c6ef15a74f707e924d5961d636e7e911e1d4494804f56ca2dccead: Status 404 returned error can't find the container with id 8138a93b86c6ef15a74f707e924d5961d636e7e911e1d4494804f56ca2dccead Nov 02 16:00:24.123715 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:24.123697 4469 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 16:00:24.659181 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:24.659146 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" event={"ID":"b03c8af7-165b-4fe1-95d2-ed27c77d22f1","Type":"ContainerStarted","Data":"8138a93b86c6ef15a74f707e924d5961d636e7e911e1d4494804f56ca2dccead"} Nov 02 16:00:25.050162 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.050124 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x"] Nov 02 16:00:25.072463 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.072432 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2fv4x"] Nov 02 16:00:25.237496 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.237421 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444bf30f-2c11-427c-bd7c-f20b760e2537" path="/var/lib/kubelet/pods/444bf30f-2c11-427c-bd7c-f20b760e2537/volumes" Nov 02 16:00:25.666410 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.666324 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" event={"ID":"b03c8af7-165b-4fe1-95d2-ed27c77d22f1","Type":"ContainerStarted","Data":"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652"} Nov 02 16:00:25.666819 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.666454 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:25.785915 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:25.785859 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" podStartSLOduration=1.9428321689999999 podStartE2EDuration="2.785843556s" totalImagesPullingTime="843.011387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:25.783886054 +0000 UTC m=+3828.667488573" watchObservedRunningTime="2026-11-02 16:00:25.785843556 +0000 UTC m=+3828.669446074" Nov 02 16:00:28.905365 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:28.905332 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:30.333093 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:30.333061 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:31.068117 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.068079 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:31.272213 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.272178 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:31.586274 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.586241 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:31.587068 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.586439 4469 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" podUID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" containerName="maas-api" containerID="cri-o://2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652" gracePeriod=30 Nov 02 16:00:31.593464 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.593444 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:31.635586 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.635557 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd"] Nov 02 16:00:31.639317 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.639299 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.690441 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.690406 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd"] Nov 02 16:00:31.728786 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.728744 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac97a06-9c07-4983-9a15-7683515c707d-openshift-service-ca\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.728965 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.728809 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-metrics-certs\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.728965 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.728913 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-maas-api-tls\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.729044 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.729013 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs8w\" (UniqueName: \"kubernetes.io/projected/eac97a06-9c07-4983-9a15-7683515c707d-kube-api-access-mhs8w\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.829980 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.829952 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac97a06-9c07-4983-9a15-7683515c707d-openshift-service-ca\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.830117 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.830000 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-metrics-certs\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.830117 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.830024 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-maas-api-tls\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.830117 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.830058 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs8w\" (UniqueName: \"kubernetes.io/projected/eac97a06-9c07-4983-9a15-7683515c707d-kube-api-access-mhs8w\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.830673 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.830648 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac97a06-9c07-4983-9a15-7683515c707d-openshift-service-ca\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.832368 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.832348 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-metrics-certs\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.832447 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.832377 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac97a06-9c07-4983-9a15-7683515c707d-maas-api-tls\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.852743 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.852689 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:31.890402 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.890376 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs8w\" (UniqueName: \"kubernetes.io/projected/eac97a06-9c07-4983-9a15-7683515c707d-kube-api-access-mhs8w\") pod \"maas-api-d95f6c575-mb8qd\" (UID: \"eac97a06-9c07-4983-9a15-7683515c707d\") " pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:31.931126 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.931099 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\") pod \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " Nov 02 16:00:31.931253 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.931184 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\" (UniqueName: \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\") pod \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " Nov 02 16:00:31.931332 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.931252 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\") pod \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " Nov 02 16:00:31.931332 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.931273 4469 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\") pod \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\" (UID: \"b03c8af7-165b-4fe1-95d2-ed27c77d22f1\") " Nov 02 16:00:31.931652 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.931616 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca" pod "b03c8af7-165b-4fe1-95d2-ed27c77d22f1" (UID: "b03c8af7-165b-4fe1-95d2-ed27c77d22f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:00:31.933249 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.933217 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls" pod "b03c8af7-165b-4fe1-95d2-ed27c77d22f1" (UID: "b03c8af7-165b-4fe1-95d2-ed27c77d22f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:00:31.933249 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.933240 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs" pod "b03c8af7-165b-4fe1-95d2-ed27c77d22f1" (UID: "b03c8af7-165b-4fe1-95d2-ed27c77d22f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:00:31.933367 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.933287 4469 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt" pod "b03c8af7-165b-4fe1-95d2-ed27c77d22f1" (UID: "b03c8af7-165b-4fe1-95d2-ed27c77d22f1"). InnerVolumeSpecName "kube-api-access-w62xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:00:31.990581 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:31.990548 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:32.032408 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.032374 4469 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-metrics-certs\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:32.032408 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.032406 4469 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62xt\" (UniqueName: \"kubernetes.io/projected/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-kube-api-access-w62xt\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:32.032600 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.032417 4469 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-openshift-service-ca\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:32.032600 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.032427 4469 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03c8af7-165b-4fe1-95d2-ed27c77d22f1-maas-api-tls\") on node \"ip-10-0-135-103.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 16:00:32.366386 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.366354 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd"] Nov 02 16:00:32.367477 ip-10-0-135-103 kubenswrapper[4469]: W1102 16:00:32.367449 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac97a06_9c07_4983_9a15_7683515c707d.slice/crio-8ef62b9e0ee9d4253bb559533bf5fddf4e202cb3a383e91a45a341135fef92e9 WatchSource:0}: Error finding container 8ef62b9e0ee9d4253bb559533bf5fddf4e202cb3a383e91a45a341135fef92e9: Status 404 returned error can't find the container with id 8ef62b9e0ee9d4253bb559533bf5fddf4e202cb3a383e91a45a341135fef92e9 Nov 02 16:00:32.705399 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.705325 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" event={"ID":"eac97a06-9c07-4983-9a15-7683515c707d","Type":"ContainerStarted","Data":"8ef62b9e0ee9d4253bb559533bf5fddf4e202cb3a383e91a45a341135fef92e9"} Nov 02 16:00:32.706923 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.706895 4469 generic.go:356] "Generic (PLEG): container finished" podID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" containerID="2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652" exitCode=0 Nov 02 16:00:32.707077 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.706938 4469 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" Nov 02 16:00:32.707077 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.706956 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" event={"ID":"b03c8af7-165b-4fe1-95d2-ed27c77d22f1","Type":"ContainerDied","Data":"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652"} Nov 02 16:00:32.707077 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.707003 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88df9b747-w4c75" event={"ID":"b03c8af7-165b-4fe1-95d2-ed27c77d22f1","Type":"ContainerDied","Data":"8138a93b86c6ef15a74f707e924d5961d636e7e911e1d4494804f56ca2dccead"} Nov 02 16:00:32.707077 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.707032 4469 scope.go:122] "RemoveContainer" containerID="2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652" Nov 02 16:00:32.716674 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.716644 4469 scope.go:122] "RemoveContainer" containerID="2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652" Nov 02 16:00:32.716971 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:00:32.716952 4469 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652\": container with ID starting with 2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652 not found: ID does not exist" containerID="2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652" Nov 02 16:00:32.717045 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.716979 4469 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652"} err="failed to get container status \"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652\": rpc error: code = NotFound desc = could not find container \"2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652\": container with ID starting with 2b6cfe599806f8b6132041940e4f9d520e7326ffec6d1e0311e63b1df3855652 not found: ID does not exist" Nov 02 16:00:32.874715 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.874658 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:32.877054 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:32.877025 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-88df9b747-w4c75"] Nov 02 16:00:33.240272 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:33.240246 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" path="/var/lib/kubelet/pods/b03c8af7-165b-4fe1-95d2-ed27c77d22f1/volumes" Nov 02 16:00:33.713994 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:33.713958 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" event={"ID":"eac97a06-9c07-4983-9a15-7683515c707d","Type":"ContainerStarted","Data":"0d5ca17f1cddf278f421e0bb31239f10e41eb89eb841076a03a7f419bb3ac11e"} Nov 02 16:00:33.714464 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:33.714089 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:33.834384 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:33.834329 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" podStartSLOduration=2.013740805 podStartE2EDuration="2.834315777s" totalImagesPullingTime="820.574972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:33.833905338 +0000 UTC m=+3836.717507858" watchObservedRunningTime="2026-11-02 16:00:33.834315777 +0000 UTC m=+3836.717918293" Nov 02 16:00:34.903816 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:34.903770 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:35.117242 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:35.117207 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:36.185706 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:00:36.185669 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:00:37.440434 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:37.440396 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:37.646869 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:37.646831 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:38.108682 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:38.108648 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:38.319169 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:38.319077 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:38.868148 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:38.868110 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:39.072784 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:39.072747 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:39.298555 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:39.298525 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:39.748846 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:39.747038 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d95f6c575-mb8qd" Nov 02 16:00:44.068904 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:44.068858 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:47.073777 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:47.067655 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:00:48.955154 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:48.955099 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:53.750096 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:53.750045 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:55.377568 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:55.377536 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:58.973510 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:00:58.973458 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:03.755337 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:03.755206 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:06.184908 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:01:06.184874 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:01:08.559636 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:08.559603 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:13.352743 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:13.352705 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:14.679212 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:14.679180 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:17.588382 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:17.588342 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:18.134675 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:18.134642 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:18.338624 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:18.338596 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:22.069056 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:22.069018 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:01:22.123319 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:22.123193 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:22.384278 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:22.384192 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:27.748316 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:27.748281 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.064144 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:29.064106 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.267051 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:29.267017 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.704061 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:29.704023 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.920045 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:29.920012 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:33.970094 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:33.970059 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:34.171679 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:34.171640 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:36.185369 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:01:36.185321 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:01:37.193694 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:37.193657 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:37.394301 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:37.394246 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:39.229434 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:39.229401 4469 scope.go:122] "RemoveContainer" containerID="f384418b54ba0e4fe54c2c25ede9d78a4c1ba4bc285fb898e46bb053d48fa35f" Nov 02 16:01:39.238979 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:39.238955 4469 scope.go:122] "RemoveContainer" containerID="5a547171b00a0b9d53f52d2ea08df60a89a60c3e2da7ac5f42864ae6f71f57cf" Nov 02 16:01:40.101704 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:40.101641 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:40.310505 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:40.310452 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:40.525101 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:40.525059 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:45.346242 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:45.346208 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:48.076304 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:48.076269 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:01:49.978965 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:49.978927 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:54.840004 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:54.839943 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:55.723469 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:55.723436 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:58.795077 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:01:58.795044 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:00.363652 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:00.363616 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:04.968948 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:04.968905 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:06.067313 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:06.067278 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt"] Nov 02 16:02:06.116284 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:06.116255 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fab16a-dxbkt"] Nov 02 16:02:06.185362 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:02:06.185328 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:02:07.239753 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:07.239719 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf" path="/var/lib/kubelet/pods/dacc1b2c-4ddc-46d0-a85e-d13d49b7fedf/volumes" Nov 02 16:02:09.883386 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:09.883329 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:14.650563 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:14.650526 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:15.015693 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:15.015666 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:17.139030 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:17.138988 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:17.348788 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:17.348749 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:19.465939 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:19.465910 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:24.282980 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:24.282925 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:29.180009 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:29.179977 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:33.944474 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:33.944430 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:36.185295 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:02:36.185259 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:02:38.758866 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:38.758790 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:39.590664 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:39.590629 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:43.549518 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:43.549480 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:48.361643 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:48.361604 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:53.328013 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:53.327978 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:56.072764 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:56.072716 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:57.854537 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:02:57.854495 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:02.642385 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:02.642352 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:06.185281 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:03:06.185246 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:03:07.445963 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:07.445911 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:12.249987 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:12.249933 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:14.518709 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:14.518666 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:14.719940 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:14.719902 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:15.422503 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:15.422469 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:17.042665 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:17.042631 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:21.992865 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:21.992634 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:26.953982 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:26.953933 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:31.738796 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:31.738746 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:31.951771 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:31.951731 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:32.159595 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:32.159499 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:36.185425 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:03:36.185391 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:03:38.993954 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:38.993912 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:39.200598 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:39.200558 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:39.267990 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:39.267915 4469 scope.go:122] "RemoveContainer" containerID="1cf5abe7f60f19eb461b96ab6415a4e1cf7e097731fff5fa169b7135f0a0d481" Nov 02 16:03:39.276244 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:39.276225 4469 scope.go:122] "RemoveContainer" containerID="0571d46da8ce2285974445dac60fb007e696430080f4110045825f12c50c2414" Nov 02 16:03:39.936749 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:39.936712 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:41.006730 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:41.006676 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:42.506404 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:42.506367 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:42.791300 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:42.791216 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:43.567950 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:43.567911 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:46.053886 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:46.053848 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:03:49.614918 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:49.614870 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:49.815497 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:49.815466 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:50.032278 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:50.032245 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:56.430748 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:56.430711 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:57.856437 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:03:57.856403 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:04:06.184684 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:04:06.184646 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:04:09.058312 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:09.058275 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m"] Nov 02 16:04:09.107499 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:09.107468 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8kc8m"] Nov 02 16:04:09.238545 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:09.238507 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8402133-4e22-4864-ae24-8ae617e89425" path="/var/lib/kubelet/pods/a8402133-4e22-4864-ae24-8ae617e89425/volumes" Nov 02 16:04:09.482192 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:09.482161 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:04:15.720064 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:15.720029 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:30.573068 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:30.573034 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:04:31.047842 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:31.047794 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99"] Nov 02 16:04:31.097107 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:31.097072 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbb99"] Nov 02 16:04:31.237523 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:31.237490 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3438add3-359e-407a-95a5-6343f09fe688" path="/var/lib/kubelet/pods/3438add3-359e-407a-95a5-6343f09fe688/volumes" Nov 02 16:04:36.193322 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:04:36.193285 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:04:40.278693 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:40.278654 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:56.778118 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:04:56.778085 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:02.539437 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:02.539405 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:03.049552 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:03.049518 4469 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n"] Nov 02 16:05:03.100290 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:03.100256 4469 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0cb807-zqx8n"] Nov 02 16:05:03.240344 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:03.240002 4469 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a75b45e-f8be-49ca-9521-1c278c6486ae" path="/var/lib/kubelet/pods/2a75b45e-f8be-49ca-9521-1c278c6486ae/volumes" Nov 02 16:05:03.676273 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:03.676241 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pvcb2"] Nov 02 16:05:05.898484 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:05.898452 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:06.185301 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:05:06.185219 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:05:14.945263 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:14.945227 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-s9wlq_af0b1a77-b60d-4066-8bb4-2408402d3522/manager/0.log" Nov 02 16:05:16.072032 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:16.072002 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h92jq_1cb6d8c6-fde1-4eee-b43e-0efb1c855907/server/0.log" Nov 02 16:05:17.350144 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:17.350110 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d95f6c575-mb8qd_eac97a06-9c07-4983-9a15-7683515c707d/maas-api/0.log" Nov 02 16:05:20.879751 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:20.879722 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qt7d_b0a3e2cd-6a7f-498d-9b05-3451e1df5d29/kuadrant-console-plugin/0.log" Nov 02 16:05:21.141837 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:21.141734 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qkfsj_f0c08a0e-075e-45fa-b789-cbcb33c3ffde/manager/0.log" Nov 02 16:05:21.642074 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:21.642040 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pvcb2_d38cccc7-83e2-4d4f-a826-7fbef4a61ac6/limitador/0.log" Nov 02 16:05:22.938222 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:22.938190 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rm9wv_f055c6fc-cc3f-40df-a429-2a4d6ed481a7/discovery/0.log" Nov 02 16:05:23.052104 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:23.052071 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:23.206466 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:23.206395 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-q7dr4_d5f80fb2-d52b-4327-87dc-a24f1e6da3ae/kube-auth-proxy/0.log" Nov 02 16:05:23.925776 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:23.925736 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-56p7g_f100f692-1c9f-426b-9292-5779f27d4558/istio-proxy/0.log" Nov 02 16:05:24.379377 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:24.379346 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5l5bb_7d428092-3abe-4a4c-a16d-2de2a1c5fe6a/istio-proxy/0.log" Nov 02 16:05:24.596760 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:24.596720 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6ql9n_1b337a30-b105-43bc-820c-0836dfd3dfd8/istio-proxy/0.log" Nov 02 16:05:26.249121 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:26.249088 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-74wfk_e8d20065-2c84-4fbf-8cca-39aaf998c068/payload-pre-processing/0.log" Nov 02 16:05:26.456542 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:26.456504 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:05:28.135670 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:28.135637 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fpcc4_fff589f3-5f1a-4549-8dcb-fa70077561c6/main/0.log" Nov 02 16:05:29.023641 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:29.023614 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm_bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e/main/0.log" Nov 02 16:05:29.243250 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:29.243222 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb_25f791a3-de1e-4890-943a-ea5230fe6268/main/0.log" Nov 02 16:05:36.186285 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:05:36.186243 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:05:39.304775 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:39.304741 4469 scope.go:122] "RemoveContainer" containerID="f185bbd18d75b737e95acb76f3a4334a3a1fc31f6dbf547a287f6201eb79d422" Nov 02 16:05:39.315384 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:39.315315 4469 scope.go:122] "RemoveContainer" containerID="db3ea7cac2337d107309810a5f3361400bb2397bcdf28079817536ec5fa60848" Nov 02 16:05:40.072306 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:40.072276 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-s9wlq_af0b1a77-b60d-4066-8bb4-2408402d3522/manager/0.log" Nov 02 16:05:40.633324 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:40.633291 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:41.095491 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:41.095465 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h92jq_1cb6d8c6-fde1-4eee-b43e-0efb1c855907/server/0.log" Nov 02 16:05:42.174088 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:42.174059 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d95f6c575-mb8qd_eac97a06-9c07-4983-9a15-7683515c707d/maas-api/0.log" Nov 02 16:05:45.488314 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:45.488287 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qt7d_b0a3e2cd-6a7f-498d-9b05-3451e1df5d29/kuadrant-console-plugin/0.log" Nov 02 16:05:45.710730 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:45.710698 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qkfsj_f0c08a0e-075e-45fa-b789-cbcb33c3ffde/manager/0.log" Nov 02 16:05:46.176712 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:46.176680 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pvcb2_d38cccc7-83e2-4d4f-a826-7fbef4a61ac6/limitador/0.log" Nov 02 16:05:47.283990 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:47.283962 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rm9wv_f055c6fc-cc3f-40df-a429-2a4d6ed481a7/discovery/0.log" Nov 02 16:05:47.599644 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:47.599572 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-q7dr4_d5f80fb2-d52b-4327-87dc-a24f1e6da3ae/kube-auth-proxy/0.log" Nov 02 16:05:48.337838 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:48.337806 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-56p7g_f100f692-1c9f-426b-9292-5779f27d4558/istio-proxy/0.log" Nov 02 16:05:48.900926 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:48.900894 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5l5bb_7d428092-3abe-4a4c-a16d-2de2a1c5fe6a/istio-proxy/0.log" Nov 02 16:05:49.152885 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:49.152789 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6ql9n_1b337a30-b105-43bc-820c-0836dfd3dfd8/istio-proxy/0.log" Nov 02 16:05:50.856584 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:50.856555 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-74wfk_e8d20065-2c84-4fbf-8cca-39aaf998c068/payload-pre-processing/0.log" Nov 02 16:05:51.085967 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:51.085937 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:05:52.535526 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:52.535501 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fpcc4_fff589f3-5f1a-4549-8dcb-fa70077561c6/main/0.log" Nov 02 16:05:53.484993 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:53.484963 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nk5jm_bb8c5e76-d6ba-48b8-a50b-9e19d4c4860e/main/0.log" Nov 02 16:05:53.725069 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:53.725039 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8jskb_25f791a3-de1e-4890-943a-ea5230fe6268/main/0.log" Nov 02 16:05:57.133990 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:57.133953 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:57.788087 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:05:57.788055 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:06.185163 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:06:06.185124 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:06:16.331215 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:16.331176 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:17.409041 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:17.409008 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:21.065019 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:21.064931 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:21.558396 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:21.558366 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:06:21.775279 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:21.775248 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:06:36.185599 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:06:36.185558 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:06:39.336441 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:39.336407 4469 scope.go:122] "RemoveContainer" containerID="393d78a8e2f6e76a20e38850a80c6c0cfb78b664d570aa591d65f58aada69e5d" Nov 02 16:06:40.990041 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:06:40.990010 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:06.185748 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:07:06.185708 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:07:08.922521 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:08.922493 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:07:09.149359 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:09.149331 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j4tg5_02cf8231-7c60-4b6f-bc1d-07dbfc48a57a/payload-processing/0.log" Nov 02 16:07:09.579596 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:09.579560 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:12.123213 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:12.123160 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:19.659469 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:19.659432 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tzjrs_09413fd8-4cfb-4e13-adb8-c8a58533ad5e/global-pull-secret-syncer/0.log" Nov 02 16:07:21.408303 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:21.408255 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:21.593379 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:21.593344 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" Nov 02 16:07:23.429559 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:23.429525 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-103.us-west-1.compute.internal_5f06372767f35a0b77d37ba2e829e38f/haproxy/0.log" Nov 02 16:07:29.061479 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:29.061447 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8qt7d_b0a3e2cd-6a7f-498d-9b05-3451e1df5d29/kuadrant-console-plugin/0.log" Nov 02 16:07:29.555907 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:29.555879 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qkfsj_f0c08a0e-075e-45fa-b789-cbcb33c3ffde/manager/0.log" Nov 02 16:07:30.147941 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:30.147916 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pvcb2_d38cccc7-83e2-4d4f-a826-7fbef4a61ac6/limitador/0.log" Nov 02 16:07:36.185232 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:07:36.185191 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:07:36.769289 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:36.769260 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q4rx4_eeb0b8b6-3e24-4b17-b733-2ff2d194a48f/kube-state-metrics/0.log" Nov 02 16:07:37.180496 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:37.180421 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q4rx4_eeb0b8b6-3e24-4b17-b733-2ff2d194a48f/kube-rbac-proxy-main/0.log" Nov 02 16:07:37.596449 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:37.596418 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q4rx4_eeb0b8b6-3e24-4b17-b733-2ff2d194a48f/kube-rbac-proxy-self/0.log" Nov 02 16:07:41.321001 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:41.320962 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:41.363415 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:41.363391 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zflnb_611bc632-ddb5-4d92-a2af-9ee53d600c41/node-exporter/0.log" Nov 02 16:07:41.777854 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:41.777819 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zflnb_611bc632-ddb5-4d92-a2af-9ee53d600c41/kube-rbac-proxy/0.log" Nov 02 16:07:42.195609 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:42.195534 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zflnb_611bc632-ddb5-4d92-a2af-9ee53d600c41/init-textfile/0.log" Nov 02 16:07:42.628031 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:42.628005 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6gxdj_c0071695-75e8-417e-a4c3-9ef622a53c5e/kube-rbac-proxy-main/0.log" Nov 02 16:07:43.042818 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:43.042775 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6gxdj_c0071695-75e8-417e-a4c3-9ef622a53c5e/kube-rbac-proxy-self/0.log" Nov 02 16:07:43.457840 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:43.457738 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6gxdj_c0071695-75e8-417e-a4c3-9ef622a53c5e/openshift-state-metrics/0.log" Nov 02 16:07:43.884667 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:43.884639 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/prometheus/0.log" Nov 02 16:07:44.299231 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:44.299200 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/config-reloader/0.log" Nov 02 16:07:44.717766 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:44.717738 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/thanos-sidecar/0.log" Nov 02 16:07:45.132962 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:45.132891 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/kube-rbac-proxy-web/0.log" Nov 02 16:07:45.549139 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:45.549111 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/kube-rbac-proxy/0.log" Nov 02 16:07:45.969545 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:45.969519 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/kube-rbac-proxy-thanos/0.log" Nov 02 16:07:46.386148 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:46.386077 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01f9099a-203c-4431-96e0-8c34baf619ae/init-config-reloader/0.log" Nov 02 16:07:48.060103 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.060068 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85b96b6b66-ck9sh_ead863a6-a542-46bb-828e-6fe9ebba5fa5/telemeter-client/0.log" Nov 02 16:07:48.236990 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.236954 4469 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr"] Nov 02 16:07:48.237388 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.237375 4469 cpu_manager.go:414] "containerMap: removing container" podUID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" containerName="maas-api" Nov 02 16:07:48.237430 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.237389 4469 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" containerName="maas-api" Nov 02 16:07:48.237482 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.237476 4469 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03c8af7-165b-4fe1-95d2-ed27c77d22f1" containerName="maas-api" Nov 02 16:07:48.240368 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.240348 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.292549 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.292523 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjv5q\"/\"openshift-service-ca.crt\"" Nov 02 16:07:48.292708 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.292660 4469 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjv5q\"/\"kube-root-ca.crt\"" Nov 02 16:07:48.342291 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.342205 4469 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gjv5q\"/\"default-dockercfg-2qgwv\"" Nov 02 16:07:48.344130 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.344093 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr"] Nov 02 16:07:48.375915 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.375887 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrk7\" (UniqueName: \"kubernetes.io/projected/2fbf5023-24f5-4311-8e37-3906201b9612-kube-api-access-vbrk7\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.376038 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.375919 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-proc\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.376038 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.375948 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-sys\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.376158 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.376136 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-lib-modules\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.376196 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.376189 4469 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-podres\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.474112 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.474077 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85b96b6b66-ck9sh_ead863a6-a542-46bb-828e-6fe9ebba5fa5/reload/0.log" Nov 02 16:07:48.477388 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477368 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-lib-modules\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477454 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477416 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-podres\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477492 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477453 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrk7\" (UniqueName: \"kubernetes.io/projected/2fbf5023-24f5-4311-8e37-3906201b9612-kube-api-access-vbrk7\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477492 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477471 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-proc\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477602 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477492 4469 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-sys\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477602 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477503 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-lib-modules\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477602 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477562 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-sys\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477602 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477572 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-proc\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.477602 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.477573 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2fbf5023-24f5-4311-8e37-3906201b9612-podres\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.538170 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.538139 4469 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrk7\" (UniqueName: \"kubernetes.io/projected/2fbf5023-24f5-4311-8e37-3906201b9612-kube-api-access-vbrk7\") pod \"perf-node-gather-daemonset-hrvhr\" (UID: \"2fbf5023-24f5-4311-8e37-3906201b9612\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.553096 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.553075 4469 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:48.893312 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.893243 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85b96b6b66-ck9sh_ead863a6-a542-46bb-828e-6fe9ebba5fa5/kube-rbac-proxy/0.log" Nov 02 16:07:48.924524 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:48.924503 4469 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr"] Nov 02 16:07:48.927009 ip-10-0-135-103 kubenswrapper[4469]: W1102 16:07:48.926986 4469 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2fbf5023_24f5_4311_8e37_3906201b9612.slice/crio-940eac3c54166282c1ec8c0c56a5e23c50ee2495946e9cd681e9debdd6e04e82 WatchSource:0}: Error finding container 940eac3c54166282c1ec8c0c56a5e23c50ee2495946e9cd681e9debdd6e04e82: Status 404 returned error can't find the container with id 940eac3c54166282c1ec8c0c56a5e23c50ee2495946e9cd681e9debdd6e04e82 Nov 02 16:07:49.918722 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:49.918645 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" event={"ID":"2fbf5023-24f5-4311-8e37-3906201b9612","Type":"ContainerStarted","Data":"300345ef4dd8eb581ffe9f1ac5c0c768a0dfbe430c584d3a030593f3857419cc"} Nov 02 16:07:49.918722 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:49.918699 4469 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" event={"ID":"2fbf5023-24f5-4311-8e37-3906201b9612","Type":"ContainerStarted","Data":"940eac3c54166282c1ec8c0c56a5e23c50ee2495946e9cd681e9debdd6e04e82"} Nov 02 16:07:49.919166 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:49.918735 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:50.051088 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:50.051036 4469 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" podStartSLOduration=2.051021097 podStartE2EDuration="2.051021097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:07:50.050038774 +0000 UTC m=+4272.933641293" watchObservedRunningTime="2026-11-02 16:07:50.051021097 +0000 UTC m=+4272.934623616" Nov 02 16:07:55.933250 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:55.933218 4469 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-hrvhr" Nov 02 16:07:59.771085 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:07:59.771051 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-4crsd_311c9cd3-8b65-4c41-82b7-f2d89f8dadc9/download-server/0.log" Nov 02 16:08:02.477041 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:02.477008 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:06.185778 ip-10-0-135-103 kubenswrapper[4469]: E1102 16:08:06.185744 4469 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4prgg" containerName="extractor" Nov 02 16:08:09.633948 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:09.633916 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7g64_f473c9a3-09d1-407c-be89-b00857fd2f56/dns/0.log" Nov 02 16:08:10.047619 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:10.047591 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w7g64_f473c9a3-09d1-407c-be89-b00857fd2f56/kube-rbac-proxy/0.log" Nov 02 16:08:10.464197 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:10.464173 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7x9dz_2cdb8471-d93e-4750-9aef-984d1a738d4a/dns-node-resolver/0.log" Nov 02 16:08:14.083234 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:14.083202 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dfdbf569-5d8rw_9075214a-7dfa-44fd-9156-bb672e74a624/registry/0.log" Nov 02 16:08:15.048762 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:15.048736 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzr9k_0959e584-488f-4332-a378-b927ac207e49/node-ca/0.log" Nov 02 16:08:21.040320 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:21.040290 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rm9wv_f055c6fc-cc3f-40df-a429-2a4d6ed481a7/discovery/0.log" Nov 02 16:08:21.390904 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:21.390833 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-q7dr4_d5f80fb2-d52b-4327-87dc-a24f1e6da3ae/kube-auth-proxy/0.log" Nov 02 16:08:21.763290 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:21.763257 4469 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lw49d_7734be1e-a6d2-4c25-9982-3f0fbe90c330/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:22.649947 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:22.649913 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-56p7g_f100f692-1c9f-426b-9292-5779f27d4558/istio-proxy/0.log" Nov 02 16:08:23.509890 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:23.509861 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5l5bb_7d428092-3abe-4a4c-a16d-2de2a1c5fe6a/istio-proxy/0.log" Nov 02 16:08:23.929733 ip-10-0-135-103 kubenswrapper[4469]: I1102 16:08:23.929661 4469 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6ql9n_1b337a30-b105-43bc-820c-0836dfd3dfd8/istio-proxy/0.log"