Jul 16 20:31:04.433335 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 20:31:04.433345 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 20:31:04.433354 ip-10-0-131-179 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 20:31:04.433842 ip-10-0-131-179 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 20:31:14.556074 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 20:31:14.556090 ip-10-0-131-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fc2b9ccb638942fd8d23116b67ee5b1a -- Jul 16 20:33:21.383631 ip-10-0-131-179 systemd[1]: Starting Kubernetes Kubelet... Jul 16 20:33:21.770524 ip-10-0-131-179 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 20:33:21.770524 ip-10-0-131-179 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 20:33:21.770524 ip-10-0-131-179 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 20:33:21.770524 ip-10-0-131-179 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 20:33:21.770524 ip-10-0-131-179 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 20:33:21.771557 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.771464 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776632 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776654 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776658 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776663 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776666 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.776658 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776670 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776673 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776676 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776678 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776681 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776683 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776686 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776688 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776691 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776693 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776696 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776699 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776701 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776704 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776706 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776709 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776712 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776714 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776717 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776720 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.776895 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776722 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776725 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776727 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776730 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776733 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776737 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776741 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776744 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776747 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776750 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776752 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776755 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776757 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776760 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776763 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776765 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776768 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776770 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776773 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776775 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.777380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776778 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776780 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776783 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776785 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776788 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776790 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776793 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776795 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776798 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776801 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776804 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776806 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776809 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776811 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776814 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776819 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776823 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776826 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776829 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.777854 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776831 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776834 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776837 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776839 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776842 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776845 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776847 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776851 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776854 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776856 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776859 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776862 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776864 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776866 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776869 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776872 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776874 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776876 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776879 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776882 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.778336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776884 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776887 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776889 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776892 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776896 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776899 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776902 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776905 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776907 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776913 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776916 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776920 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776923 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776925 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.776928 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778344 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778352 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778355 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778358 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.778803 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778361 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778364 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778367 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778370 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778372 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778375 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778377 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778380 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778382 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778385 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778387 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778389 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778392 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778396 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778398 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778402 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778404 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778407 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778410 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778413 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.779297 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778416 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778418 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778421 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778423 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778426 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778428 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778431 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778433 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778436 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778438 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778441 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778443 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778446 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778448 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778451 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778454 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778456 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778459 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778461 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778463 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.779804 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778466 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778468 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778470 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778473 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778476 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778478 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778481 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778485 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778488 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778490 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778493 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778511 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778516 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778519 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778522 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778525 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778530 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778533 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778536 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.780288 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778540 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778544 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778548 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778551 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778553 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778556 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778559 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778562 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778566 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778569 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778572 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778575 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778577 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778580 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778583 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778585 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778588 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778590 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778592 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.780765 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778595 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778598 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778602 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778604 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778607 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778609 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778612 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778614 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778617 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778619 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778622 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778624 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778627 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778629 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778632 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778634 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.778637 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779027 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779032 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779036 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.781238 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779038 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779041 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779043 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779046 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779049 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779051 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779054 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779056 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779059 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779061 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779064 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779066 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779069 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779072 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779076 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779078 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779081 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779084 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779086 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.781731 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779089 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779092 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779094 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779097 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779099 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779102 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779104 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779106 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779109 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779111 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779113 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779116 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779118 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779121 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779123 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779125 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779128 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779130 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779133 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779138 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.782181 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779141 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779143 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779146 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779149 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779151 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779154 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779156 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779160 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779163 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779165 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779167 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779170 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779172 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779175 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779177 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779179 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779182 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779184 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779187 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779189 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.782672 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779191 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779194 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779196 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779198 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779204 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779207 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779209 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779212 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779214 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779217 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779220 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779234 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779237 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779240 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779242 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779245 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779247 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779250 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779252 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779255 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.783148 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779258 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779260 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779263 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779265 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779268 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779270 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779274 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779277 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779281 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779284 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779288 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779292 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779296 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779313 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779317 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779320 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.779324 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779406 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779418 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779427 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 20:33:21.783684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779446 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779452 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779457 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779464 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779468 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779471 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779474 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779478 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779481 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779484 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779487 2564 flags.go:64] FLAG: --cgroup-root="" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779490 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779493 2564 flags.go:64] FLAG: --client-ca-file="" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779497 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779500 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779505 2564 flags.go:64] FLAG: --cluster-domain="" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779508 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779511 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779514 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779517 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779521 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779524 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779527 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 20:33:21.784159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779530 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779533 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779536 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779540 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779543 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779546 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779551 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779554 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779556 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779559 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779562 2564 flags.go:64] FLAG: --enable-server="true" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779565 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779571 2564 flags.go:64] FLAG: --event-burst="100" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779574 2564 flags.go:64] FLAG: --event-qps="50" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779577 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779580 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779583 2564 flags.go:64] FLAG: --eviction-hard="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779587 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779590 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779593 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779596 2564 flags.go:64] FLAG: --eviction-soft="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779599 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779602 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779605 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779608 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 20:33:21.784716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779611 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779614 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779617 2564 flags.go:64] FLAG: --feature-gates="" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779621 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779623 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779626 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779629 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779632 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779635 2564 flags.go:64] FLAG: --help="false" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779638 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-179.ec2.internal" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779641 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779643 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779648 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779651 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779654 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779657 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779659 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779662 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779665 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779669 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779672 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779675 2564 flags.go:64] FLAG: --kube-reserved="" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779678 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779681 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 20:33:21.785347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779684 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779687 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779690 2564 flags.go:64] FLAG: --lock-file="" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779692 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779695 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779698 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779703 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779708 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779711 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779714 2564 flags.go:64] FLAG: --logging-format="text" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779717 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779720 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779723 2564 flags.go:64] FLAG: --manifest-url="" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779725 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779730 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779733 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779737 2564 flags.go:64] FLAG: --max-pods="110" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779740 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779743 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779746 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779750 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779752 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779756 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779758 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779766 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 20:33:21.785975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779769 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779773 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779777 2564 flags.go:64] FLAG: --pod-cidr="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779780 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779786 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779788 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779792 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779794 2564 flags.go:64] FLAG: --port="10250" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779797 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779800 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001e9b64c3a484d44" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779803 2564 flags.go:64] FLAG: --qos-reserved="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779806 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779810 2564 flags.go:64] FLAG: --register-node="true" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779813 2564 flags.go:64] FLAG: --register-with-taints="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779816 2564 flags.go:64] FLAG: --registry-burst="10" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779820 2564 flags.go:64] FLAG: --registry-qps="5" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779823 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779825 2564 flags.go:64] FLAG: --reserved-memory="" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779829 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779832 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779835 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779838 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779841 2564 flags.go:64] FLAG: --runonce="false" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779843 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779846 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 20:33:21.786625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779849 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779852 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779854 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779858 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779861 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779864 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779867 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779870 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779873 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779877 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779880 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779882 2564 flags.go:64] FLAG: --system-cgroups="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779885 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779890 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779893 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779896 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779901 2564 flags.go:64] FLAG: --tls-min-version="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779903 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779906 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779909 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779911 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779914 2564 flags.go:64] FLAG: --v="2" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779919 2564 flags.go:64] FLAG: --version="false" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779923 2564 flags.go:64] FLAG: --vmodule="" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779927 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 20:33:21.787287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.779930 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780063 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780067 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780070 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780073 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780075 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780078 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780081 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780083 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780086 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780090 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780093 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780096 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780099 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780101 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780104 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780108 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780111 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780113 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.787925 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780116 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780118 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780121 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780123 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780126 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780129 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780131 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780134 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780137 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780139 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780143 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780147 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780152 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780155 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780159 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780161 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780164 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780166 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780169 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.788503 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780171 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780174 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780176 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780179 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780183 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780185 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780188 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780190 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780193 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780195 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780199 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780202 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780204 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780207 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780209 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780212 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780215 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780218 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780220 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780223 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.788979 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780225 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780227 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780231 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780233 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780235 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780238 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780241 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780244 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780246 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780248 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780251 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780253 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780257 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780260 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780263 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780266 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780270 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780273 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780276 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.789475 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780278 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780281 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780283 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780288 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780291 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780294 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780311 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780315 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780317 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780320 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780323 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780326 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780329 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780331 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780334 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780337 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780340 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780342 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780345 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780348 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.789948 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780351 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780353 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.780356 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.780361 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.787461 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.787480 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.788443 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.788453 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788520 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788526 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788530 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788534 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788536 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788539 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788542 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.790448 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788546 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788548 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788551 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788554 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788558 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788562 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788565 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788568 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788571 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788574 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788577 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788580 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788582 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788585 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788587 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788590 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788593 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788595 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788597 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.790867 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788600 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788602 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788605 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788609 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788612 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788615 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788618 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788622 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788625 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788627 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788630 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788633 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788635 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788638 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788641 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788643 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788646 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788649 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788651 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788654 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.791453 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788656 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788659 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788662 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788664 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788667 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788669 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788672 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788674 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788677 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788679 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788682 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788684 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788687 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788690 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788692 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788695 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788698 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788700 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788703 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788705 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.791975 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788708 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788711 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788713 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788716 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788718 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788721 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788724 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788726 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788729 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788732 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788734 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788737 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788739 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788742 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788744 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788746 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788749 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788751 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788754 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.792476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788756 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788759 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788761 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788763 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788766 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788768 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788771 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788773 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788777 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788780 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788784 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788787 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788789 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788792 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.788797 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 20:33:21.792942 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788943 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788949 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788952 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788955 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788958 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788961 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788963 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788966 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788969 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788971 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788973 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788976 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788979 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788981 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788985 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788989 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788993 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788996 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.788999 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789002 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 20:33:21.793353 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789004 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789007 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789009 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789012 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789014 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789017 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789019 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789022 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789030 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789033 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789036 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789039 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789042 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789045 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789048 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789052 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789056 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789060 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789063 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 20:33:21.793843 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789065 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789068 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789070 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789073 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789075 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789079 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789081 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789083 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789086 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789088 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789091 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789093 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789095 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789098 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789100 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789103 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789105 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789108 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789110 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789113 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 20:33:21.794324 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789115 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789118 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789127 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789130 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789133 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789136 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789138 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789141 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789143 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789146 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789148 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789151 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789153 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789155 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789158 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789160 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789163 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789165 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789168 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789170 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 20:33:21.794807 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789175 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789178 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789180 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789183 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789185 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789188 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789190 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789193 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789195 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789198 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789200 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789203 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789205 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789209 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789212 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789220 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789222 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789225 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789227 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 20:33:21.795331 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:21.789229 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.789235 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.789418 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.791382 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.793062 2564 server.go:1019] "Starting client certificate rotation" Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.793169 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 20:33:21.795815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.793206 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 20:33:21.796090 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.796074 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 20:33:21.801289 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.801263 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 20:33:21.810742 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.810715 2564 log.go:25] "Validated CRI v1 runtime API" Jul 16 20:33:21.812762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.812744 2564 log.go:25] "Validated CRI v1 image API" Jul 16 20:33:21.813547 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.813532 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 20:33:21.816858 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.816830 2564 fs.go:135] Filesystem UUIDs: map[49d3664a-f3e5-480b-b23d-e97ad45c2112:/dev/nvme0n1p4 60383174-ed53-4ffb-8298-8b6d1fbaf552:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 16 20:33:21.816908 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.816858 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 20:33:21.822043 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.822022 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 20:33:21.822343 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.822178 2564 manager.go:216] Machine: {Timestamp:2026-07-16 20:33:21.82074025 +0000 UTC m=+0.321820987 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097900 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bca61403e594ed825918e7a15ca44 SystemUUID:ec2bca61-403e-594e-d825-918e7a15ca44 BootID:fc2b9ccb-6389-42fd-8d23-116b67ee5b1a Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:cb:fa:70:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:cb:fa:70:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:04:b8:14:d4:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 20:33:21.822343 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.822339 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 20:33:21.822447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.822433 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 20:33:21.825379 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.825351 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 20:33:21.825543 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.825383 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 20:33:21.826634 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.826623 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 20:33:21.826711 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.826636 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 20:33:21.826711 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.826643 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 20:33:21.827336 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.827324 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 20:33:21.828531 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.828519 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 20:33:21.831105 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831090 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 16 20:33:21.831255 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831244 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 20:33:21.831310 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831280 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 16 20:33:21.831346 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831313 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 20:33:21.831346 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831336 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 20:33:21.831400 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831389 2564 kubelet.go:398] "Adding apiserver pod source" Jul 16 20:33:21.831400 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831399 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 20:33:21.831863 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831849 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 20:33:21.831895 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.831874 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 20:33:21.835343 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.835318 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 20:33:21.836747 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.836724 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 20:33:21.836843 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.836763 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 20:33:21.837404 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837390 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837413 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837421 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837426 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837435 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837440 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837448 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 20:33:21.837451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837453 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 20:33:21.837630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837461 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 20:33:21.837630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837467 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 20:33:21.837630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837475 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 20:33:21.837630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.837481 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 20:33:21.838511 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.838500 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 20:33:21.838567 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.838513 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 20:33:21.840554 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.840531 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dprpx" Jul 16 20:33:21.841684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.841666 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-179.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 20:33:21.841897 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.841873 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 20:33:21.842000 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.841961 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 20:33:21.842065 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.842011 2564 server.go:1268] "Started kubelet" Jul 16 20:33:21.845561 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.845523 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 20:33:21.846066 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.846026 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 20:33:21.846144 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.846096 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 20:33:21.846575 ip-10-0-131-179 systemd[1]: Started Kubernetes Kubelet. Jul 16 20:33:21.847958 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.847766 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 20:33:21.848762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.848725 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dprpx" Jul 16 20:33:21.850660 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.850622 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 20:33:21.851013 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.851000 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 16 20:33:21.851915 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.850978 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-179.ec2.internal.18c2df5f13f6de92 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-179.ec2.internal,UID:ip-10-0-131-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-179.ec2.internal,},FirstTimestamp:2026-07-16 20:33:21.841970834 +0000 UTC m=+0.343051551,LastTimestamp:2026-07-16 20:33:21.841970834 +0000 UTC m=+0.343051551,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-179.ec2.internal,}" Jul 16 20:33:21.855668 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.855640 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 20:33:21.855762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.855688 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 20:33:21.855762 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.855738 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:21.856197 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856175 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 20:33:21.856197 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856198 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 20:33:21.856353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856182 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 20:33:21.856353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856343 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 16 20:33:21.856353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856351 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 16 20:33:21.856517 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856501 2564 factory.go:55] Registering systemd factory Jul 16 20:33:21.856559 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856523 2564 factory.go:223] Registration of the systemd container factory successfully Jul 16 20:33:21.856798 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856783 2564 factory.go:153] Registering CRI-O factory Jul 16 20:33:21.856798 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856797 2564 factory.go:223] Registration of the crio container factory successfully Jul 16 20:33:21.856947 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856844 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 20:33:21.856947 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856866 2564 factory.go:103] Registering Raw factory Jul 16 20:33:21.856947 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.856880 2564 manager.go:1196] Started watching for new ooms in manager Jul 16 20:33:21.857354 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.857340 2564 manager.go:349] Starting recovery of all containers Jul 16 20:33:21.857882 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.857818 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:21.860486 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.860461 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-179.ec2.internal\" not found" node="ip-10-0-131-179.ec2.internal" Jul 16 20:33:21.865195 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.865030 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 20:33:21.867764 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.867743 2564 manager.go:354] Recovery completed Jul 16 20:33:21.869470 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.869435 2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 16 20:33:21.873317 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.873289 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:21.875216 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875197 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:21.875355 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875240 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:21.875355 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875256 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:21.875894 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875881 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 20:33:21.875941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875895 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 20:33:21.875941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875919 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 16 20:33:21.875941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.875925 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:21.877143 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877129 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:21.877211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877159 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:21.877211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877171 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:21.877284 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877271 2564 policy_none.go:49] "None policy: Start" Jul 16 20:33:21.877342 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877288 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 20:33:21.877376 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.877348 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 20:33:21.878386 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.878374 2564 policy_none.go:47] "Start" Jul 16 20:33:21.911005 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.910979 2564 manager.go:337] "Starting Device Plugin manager" Jul 16 20:33:21.911193 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.911020 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 20:33:21.911193 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911030 2564 server.go:87] "Starting device plugin registration server" Jul 16 20:33:21.911385 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911370 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 20:33:21.911440 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911391 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 20:33:21.911516 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911493 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 20:33:21.911858 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911599 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 20:33:21.911858 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:21.911614 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 20:33:21.912128 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.912107 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 20:33:21.912235 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:21.912159 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.011889 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.011852 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:22.013031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.012913 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:22.013031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.012948 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:22.013031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.012964 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:22.013031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.012992 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.019520 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.019496 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.019588 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.019528 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-179.ec2.internal\": node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.032821 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.032734 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 20:33:22.032821 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.032782 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 20:33:22.032821 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.032805 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 20:33:22.033003 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.032843 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 20:33:22.035322 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.035277 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:22.035917 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.035897 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.133716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.133680 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal"] Jul 16 20:33:22.133822 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.133778 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:22.135580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.135563 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:22.135651 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.135595 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:22.135651 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.135606 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:22.136638 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.136622 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.136781 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.136769 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:22.136879 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.136865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.136931 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.136896 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:22.137566 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137542 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:22.137661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137555 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:22.137661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137583 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:22.137661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137599 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:22.137661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137604 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:22.137661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.137615 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:22.138858 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.138843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.138956 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.138867 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 20:33:22.139578 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.139558 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Jul 16 20:33:22.139647 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.139590 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 20:33:22.139647 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.139599 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Jul 16 20:33:22.157924 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.157901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.158033 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.157928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.158033 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.157947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d0a72e3dbfc85402677ef718ad352e5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"2d0a72e3dbfc85402677ef718ad352e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.158033 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.158003 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-179.ec2.internal\" not found" node="ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.161865 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.161843 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-179.ec2.internal\" not found" node="ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.237161 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.237127 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.258690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.258768 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.258768 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d0a72e3dbfc85402677ef718ad352e5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"2d0a72e3dbfc85402677ef718ad352e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.258768 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d0a72e3dbfc85402677ef718ad352e5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"2d0a72e3dbfc85402677ef718ad352e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.258864 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.258864 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.258762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3cc710395b7929d32dc5b7fe48ac3bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"a3cc710395b7929d32dc5b7fe48ac3bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.338178 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.338104 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.438662 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.438617 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.459997 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.459958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.465870 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.465848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.539348 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.539288 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.640165 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.640086 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.740471 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.740430 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Jul 16 20:33:22.746303 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.746283 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:22.748697 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.748680 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:22.755983 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.755969 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.768013 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.767992 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 20:33:22.769241 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.769225 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Jul 16 20:33:22.778122 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.778101 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 20:33:22.792311 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.792276 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 20:33:22.792490 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.792467 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 20:33:22.792552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.792489 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 20:33:22.792552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.792486 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 20:33:22.792552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.792525 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 20:33:22.833715 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.833678 2564 apiserver.go:52] "Watching apiserver" Jul 16 20:33:22.850886 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.850854 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 20:33:22.851266 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.851241 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h","openshift-cluster-node-tuning-operator/tuned-h56nk","openshift-image-registry/node-ca-tfs5x","openshift-multus/multus-4zjhb","openshift-multus/multus-additional-cni-plugins-s2gq9","openshift-network-diagnostics/network-check-target-jldkm","kube-system/konnectivity-agent-mrzzd","openshift-dns/node-resolver-f7sgr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","openshift-multus/network-metrics-daemon-5dddd","openshift-network-operator/iptables-alerter-wn8jn","openshift-ovn-kubernetes/ovnkube-node-j6xlh","kube-system/global-pull-secret-syncer-vz888","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal"] Jul 16 20:33:22.852820 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.852801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.855327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.853283 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 20:28:22 +0000 UTC" deadline="2027-12-14 16:23:02.230606165 +0000 UTC" Jul 16 20:33:22.855327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.853337 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12379h49m39.377293466s" Jul 16 20:33:22.855327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.855233 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8pqsb\"" Jul 16 20:33:22.855725 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.855699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.856106 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.856075 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 20:33:22.856106 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.856088 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 20:33:22.856106 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.856092 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:22.857073 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.857056 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 20:33:22.857466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.857445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.858570 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.858553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.859688 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.859669 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.860064 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fzshc\"" Jul 16 20:33:22.860169 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860068 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.860169 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860098 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.860302 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 20:33:22.860777 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:22.860868 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.860836 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:22.860936 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860878 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 20:33:22.861017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.860973 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.861114 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861098 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 20:33:22.861181 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-dbus\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.861181 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.861181 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57063013-98eb-4fb8-a531-c9053a008579-serviceca\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptvd\" (UniqueName: \"kubernetes.io/projected/57063013-98eb-4fb8-a531-c9053a008579-kube-api-access-nptvd\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1141b-91ef-4a48-94a9-d33108ccecee-konnectivity-ca\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57063013-98eb-4fb8-a531-c9053a008579-host\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d1141b-91ef-4a48-94a9-d33108ccecee-agent-certs\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-kubelet-config\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.861402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861374 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jrftb\"" Jul 16 20:33:22.861971 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.861956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:22.862185 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.862161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-865jb\"" Jul 16 20:33:22.862285 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.862192 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 20:33:22.862285 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.862268 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 20:33:22.862527 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.862512 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.863249 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.863228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.864580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.864549 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.864677 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.864628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.864677 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.864647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nffj2\"" Jul 16 20:33:22.864800 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.864680 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.865629 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.865609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 20:33:22.865730 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.865637 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7shrg\"" Jul 16 20:33:22.865730 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.865656 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.865842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.865615 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.866069 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.866054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:22.866164 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.866143 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:22.866927 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.866909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r7drx\"" Jul 16 20:33:22.867310 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.867290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:22.867419 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.867323 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 20:33:22.867419 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.867329 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.867419 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.867366 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.868776 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.868757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.869917 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.869899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 20:33:22.870155 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.870139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l55tb\"" Jul 16 20:33:22.870155 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.870153 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.870289 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.870144 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.870956 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.870942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 20:33:22.871067 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871055 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 20:33:22.871641 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jppmm\"" Jul 16 20:33:22.871801 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 20:33:22.871839 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871814 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 20:33:22.871839 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 20:33:22.871922 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.871912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 20:33:22.893870 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.893789 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdnrc" Jul 16 20:33:22.905900 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.905321 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdnrc" Jul 16 20:33:22.957439 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.957419 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 20:33:22.961478 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57063013-98eb-4fb8-a531-c9053a008579-serviceca\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.961560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cni-binary-copy\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.961560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-multus\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.961667 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-binary-copy\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.961667 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-run\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.961767 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tjcm\" (UniqueName: \"kubernetes.io/projected/ee1d15ed-9d78-450f-9a27-ebd324a0394c-kube-api-access-7tjcm\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.961767 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-kubernetes\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.961834 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-conf\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.961834 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-netd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.961911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-modprobe-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.961911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zsv\" (UniqueName: \"kubernetes.io/projected/2cb64f02-7b87-4004-9126-347a2a912e2f-kube-api-access-97zsv\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.961992 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-var-lib-kubelet\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.961992 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-kubelet-config\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.961992 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-systemd-units\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.961989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-systemd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57063013-98eb-4fb8-a531-c9053a008579-serviceca\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.962101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-var-lib-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-kubelet-config\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-etc-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-system-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-os-release\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-netns\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-cnibin\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.962247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-device-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bd4\" (UniqueName: \"kubernetes.io/projected/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kube-api-access-m6bd4\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-lib-modules\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn8d\" (UniqueName: \"kubernetes.io/projected/dfee882d-4604-4615-b1dc-c7af28f1967f-kube-api-access-dgn8d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptvd\" (UniqueName: \"kubernetes.io/projected/57063013-98eb-4fb8-a531-c9053a008579-kube-api-access-nptvd\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-node-log\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57063013-98eb-4fb8-a531-c9053a008579-host\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.962617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-iptables-alerter-script\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57063013-98eb-4fb8-a531-c9053a008579-host\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxvp\" (UniqueName: \"kubernetes.io/projected/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-kube-api-access-txxvp\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-etc-kubernetes\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-os-release\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-netns\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cnibin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-bin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-host\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-tmp\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-multus-certs\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.962992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-system-cni-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d1141b-91ef-4a48-94a9-d33108ccecee-agent-certs\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-kubelet\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-tmp-dir\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysconfig\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-host-slash\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-config\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963455 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-socket-dir-parent\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-conf-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-hosts-file\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-registration-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-systemd\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.963695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cb64f02-7b87-4004-9126-347a2a912e2f-ovn-node-metrics-cert\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2djt\" (UniqueName: \"kubernetes.io/projected/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-kube-api-access-s2djt\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-sys\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-tuned\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-hostroot\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.963481 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-socket-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-sys-fs\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-dbus\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:22.963859 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:23.463819989 +0000 UTC m=+1.964900721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/157e505d-0be0-4b00-88aa-2d9ab9a68b15-dbus\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpjk\" (UniqueName: \"kubernetes.io/projected/23fc2b71-d828-4b25-9122-c09e1bb02140-kube-api-access-8dpjk\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-kubelet\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.963973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-slash\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-env-overrides\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.964591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-script-lib\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-k8s-cni-cncf-io\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-ovn\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-log-socket\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1141b-91ef-4a48-94a9-d33108ccecee-konnectivity-ca\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-bin\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-daemon-config\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrp4c\" (UniqueName: \"kubernetes.io/projected/c7d430ca-5628-4814-9959-e63438410d86-kube-api-access-rrp4c\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:22.965314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.964717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1141b-91ef-4a48-94a9-d33108ccecee-konnectivity-ca\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.967237 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.967208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d1141b-91ef-4a48-94a9-d33108ccecee-agent-certs\") pod \"konnectivity-agent-mrzzd\" (UID: \"e6d1141b-91ef-4a48-94a9-d33108ccecee\") " pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:22.978235 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:22.978210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptvd\" (UniqueName: \"kubernetes.io/projected/57063013-98eb-4fb8-a531-c9053a008579-kube-api-access-nptvd\") pod \"node-ca-tfs5x\" (UID: \"57063013-98eb-4fb8-a531-c9053a008579\") " pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:23.053476 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.053427 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0a72e3dbfc85402677ef718ad352e5.slice/crio-e6db0fb235bc0d3b5f318ffb35f75b0bdd8b58cd543156a692ec98964c3c54a7 WatchSource:0}: Error finding container e6db0fb235bc0d3b5f318ffb35f75b0bdd8b58cd543156a692ec98964c3c54a7: Status 404 returned error can't find the container with id e6db0fb235bc0d3b5f318ffb35f75b0bdd8b58cd543156a692ec98964c3c54a7 Jul 16 20:33:23.057036 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.057018 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 20:33:23.064604 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-host\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.064716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-tmp\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.064716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-multus-certs\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.064716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-system-cni-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.064716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-host\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.064716 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-multus-certs\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-system-cni-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-kubelet\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-tmp-dir\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysconfig\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-host-slash\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-config\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-host-slash\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysconfig\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-kubelet\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-socket-dir-parent\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.064979 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.064903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-socket-dir-parent\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-conf-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-hosts-file\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-registration-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-conf-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-hosts-file\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-tmp-dir\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-systemd\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cb64f02-7b87-4004-9126-347a2a912e2f-ovn-node-metrics-cert\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-registration-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-systemd\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.065644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2djt\" (UniqueName: \"kubernetes.io/projected/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-kube-api-access-s2djt\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-sys\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-tuned\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-hostroot\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-sys\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-socket-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-config\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.065978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-hostroot\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-socket-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-sys-fs\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpjk\" (UniqueName: \"kubernetes.io/projected/23fc2b71-d828-4b25-9122-c09e1bb02140-kube-api-access-8dpjk\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:23.066211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-kubelet\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-slash\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-slash\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-env-overrides\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-sys-fs\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-script-lib\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-k8s-cni-cncf-io\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.066805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-k8s-cni-cncf-io\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-ovn\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-log-socket\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-bin\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-log-socket\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.066963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-daemon-config\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-ovn\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrp4c\" (UniqueName: \"kubernetes.io/projected/c7d430ca-5628-4814-9959-e63438410d86-kube-api-access-rrp4c\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-bin\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-ovnkube-script-lib\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cni-binary-copy\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.067290 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.067260 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.067319 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:23.567301447 +0000 UTC m=+2.068382153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-multus\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-binary-copy\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-run\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tjcm\" (UniqueName: \"kubernetes.io/projected/ee1d15ed-9d78-450f-9a27-ebd324a0394c-kube-api-access-7tjcm\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-kubernetes\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-conf\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-netd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-modprobe-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zsv\" (UniqueName: \"kubernetes.io/projected/2cb64f02-7b87-4004-9126-347a2a912e2f-kube-api-access-97zsv\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-var-lib-kubelet\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-systemd-units\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-systemd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-kubernetes\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.067935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-var-lib-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.067984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-var-lib-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-sysctl-conf\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-cni-netd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-modprobe-d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-daemon-config\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-var-lib-kubelet\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.068690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cni-binary-copy\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-systemd\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-systemd-units\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-multus\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-run\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-etc-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-system-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-os-release\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.068970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-netns\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-system-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-run-netns\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-etc-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-os-release\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-cnibin\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-kubelet\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-device-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-cnibin\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bd4\" (UniqueName: \"kubernetes.io/projected/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kube-api-access-m6bd4\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-run-openvswitch\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-binary-copy\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-device-dir\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cb64f02-7b87-4004-9126-347a2a912e2f-env-overrides\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.069560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-lib-modules\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn8d\" (UniqueName: \"kubernetes.io/projected/dfee882d-4604-4615-b1dc-c7af28f1967f-kube-api-access-dgn8d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-multus-cni-dir\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfee882d-4604-4615-b1dc-c7af28f1967f-lib-modules\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-tmp\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-node-log\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-iptables-alerter-script\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-node-log\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.069983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxvp\" (UniqueName: \"kubernetes.io/projected/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-kube-api-access-txxvp\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfee882d-4604-4615-b1dc-c7af28f1967f-etc-tuned\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-etc-kubernetes\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-os-release\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-netns\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.070276 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-etc-kubernetes\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7d430ca-5628-4814-9959-e63438410d86-os-release\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cnibin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-cnibin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cb64f02-7b87-4004-9126-347a2a912e2f-host-run-netns\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-bin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee1d15ed-9d78-450f-9a27-ebd324a0394c-host-var-lib-cni-bin\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-iptables-alerter-script\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.071192 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.070806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d430ca-5628-4814-9959-e63438410d86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.073231 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.073204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cb64f02-7b87-4004-9126-347a2a912e2f-ovn-node-metrics-cert\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.074126 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.074107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2djt\" (UniqueName: \"kubernetes.io/projected/ddf4c0ed-54c3-4ad0-b57f-57b227239cdb-kube-api-access-s2djt\") pod \"node-resolver-f7sgr\" (UID: \"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb\") " pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.074969 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.074949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpjk\" (UniqueName: \"kubernetes.io/projected/23fc2b71-d828-4b25-9122-c09e1bb02140-kube-api-access-8dpjk\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:23.075480 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.075460 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:23.075480 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.075482 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:23.075593 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.075492 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:23.075593 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.075546 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:23.575532854 +0000 UTC m=+2.076613560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:23.077560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.077540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrp4c\" (UniqueName: \"kubernetes.io/projected/c7d430ca-5628-4814-9959-e63438410d86-kube-api-access-rrp4c\") pod \"multus-additional-cni-plugins-s2gq9\" (UID: \"c7d430ca-5628-4814-9959-e63438410d86\") " pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.081754 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.081723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tjcm\" (UniqueName: \"kubernetes.io/projected/ee1d15ed-9d78-450f-9a27-ebd324a0394c-kube-api-access-7tjcm\") pod \"multus-4zjhb\" (UID: \"ee1d15ed-9d78-450f-9a27-ebd324a0394c\") " pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.082012 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.081966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxvp\" (UniqueName: \"kubernetes.io/projected/5066c3d4-7b1d-4fec-a9ba-2fe515879e2d-kube-api-access-txxvp\") pod \"iptables-alerter-wn8jn\" (UID: \"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d\") " pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.082012 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.081994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bd4\" (UniqueName: \"kubernetes.io/projected/b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0-kube-api-access-m6bd4\") pod \"aws-ebs-csi-driver-node-wzk6h\" (UID: \"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.082327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.082308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn8d\" (UniqueName: \"kubernetes.io/projected/dfee882d-4604-4615-b1dc-c7af28f1967f-kube-api-access-dgn8d\") pod \"tuned-h56nk\" (UID: \"dfee882d-4604-4615-b1dc-c7af28f1967f\") " pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.082408 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.082346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zsv\" (UniqueName: \"kubernetes.io/projected/2cb64f02-7b87-4004-9126-347a2a912e2f-kube-api-access-97zsv\") pod \"ovnkube-node-j6xlh\" (UID: \"2cb64f02-7b87-4004-9126-347a2a912e2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.181482 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.181398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:23.191139 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.191114 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d1141b_91ef_4a48_94a9_d33108ccecee.slice/crio-9f7bd7d1d2401a77387fe80c5e99562ad444347774fbf7651b4291c9b5aef4f2 WatchSource:0}: Error finding container 9f7bd7d1d2401a77387fe80c5e99562ad444347774fbf7651b4291c9b5aef4f2: Status 404 returned error can't find the container with id 9f7bd7d1d2401a77387fe80c5e99562ad444347774fbf7651b4291c9b5aef4f2 Jul 16 20:33:23.201632 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.201602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfs5x" Jul 16 20:33:23.209924 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.209893 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57063013_98eb_4fb8_a531_c9053a008579.slice/crio-4fe2b607b83a5147844e6cb6a45b608456023479032dfcbb17ecb7f757ddb057 WatchSource:0}: Error finding container 4fe2b607b83a5147844e6cb6a45b608456023479032dfcbb17ecb7f757ddb057: Status 404 returned error can't find the container with id 4fe2b607b83a5147844e6cb6a45b608456023479032dfcbb17ecb7f757ddb057 Jul 16 20:33:23.211648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.211624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zjhb" Jul 16 20:33:23.221215 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.221188 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1d15ed_9d78_450f_9a27_ebd324a0394c.slice/crio-77866b973b15ce85d4ce0a6f777f9d1edb30b233346176bcce529facc7ab5a38 WatchSource:0}: Error finding container 77866b973b15ce85d4ce0a6f777f9d1edb30b233346176bcce529facc7ab5a38: Status 404 returned error can't find the container with id 77866b973b15ce85d4ce0a6f777f9d1edb30b233346176bcce529facc7ab5a38 Jul 16 20:33:23.240692 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.240660 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" Jul 16 20:33:23.248611 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.248582 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d430ca_5628_4814_9959_e63438410d86.slice/crio-a939c1135e670e6dfa4ca239f38befd0807ccff705805f312a9d82c0c7b34afc WatchSource:0}: Error finding container a939c1135e670e6dfa4ca239f38befd0807ccff705805f312a9d82c0c7b34afc: Status 404 returned error can't find the container with id a939c1135e670e6dfa4ca239f38befd0807ccff705805f312a9d82c0c7b34afc Jul 16 20:33:23.253475 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.253453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f7sgr" Jul 16 20:33:23.260971 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.260947 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf4c0ed_54c3_4ad0_b57f_57b227239cdb.slice/crio-32bd5fb3f4f567908710f50cf6e425150a46e6d6e7b9eeda4bf30ff36fb628cc WatchSource:0}: Error finding container 32bd5fb3f4f567908710f50cf6e425150a46e6d6e7b9eeda4bf30ff36fb628cc: Status 404 returned error can't find the container with id 32bd5fb3f4f567908710f50cf6e425150a46e6d6e7b9eeda4bf30ff36fb628cc Jul 16 20:33:23.272150 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.272125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" Jul 16 20:33:23.279940 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.279908 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56b6b6a_a490_4f8c_a5c6_c0079eea3bd0.slice/crio-81f304a21fdc91c45b38b7d32da46285e1d0b19b00b255929e8b91f95cbeaceb WatchSource:0}: Error finding container 81f304a21fdc91c45b38b7d32da46285e1d0b19b00b255929e8b91f95cbeaceb: Status 404 returned error can't find the container with id 81f304a21fdc91c45b38b7d32da46285e1d0b19b00b255929e8b91f95cbeaceb Jul 16 20:33:23.293388 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.293367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h56nk" Jul 16 20:33:23.300625 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.300599 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfee882d_4604_4615_b1dc_c7af28f1967f.slice/crio-f6c1cedb1c093c6e25284bd0787eb7f5038e44fbdc71373047da2639f39e31bf WatchSource:0}: Error finding container f6c1cedb1c093c6e25284bd0787eb7f5038e44fbdc71373047da2639f39e31bf: Status 404 returned error can't find the container with id f6c1cedb1c093c6e25284bd0787eb7f5038e44fbdc71373047da2639f39e31bf Jul 16 20:33:23.306157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.306138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wn8jn" Jul 16 20:33:23.311128 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.311101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:23.314400 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.314352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5066c3d4_7b1d_4fec_a9ba_2fe515879e2d.slice/crio-7cfc46ce13bc509601c265d59ca14eb8931830bd048c02844b3d53bef4fa1b5a WatchSource:0}: Error finding container 7cfc46ce13bc509601c265d59ca14eb8931830bd048c02844b3d53bef4fa1b5a: Status 404 returned error can't find the container with id 7cfc46ce13bc509601c265d59ca14eb8931830bd048c02844b3d53bef4fa1b5a Jul 16 20:33:23.319493 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:23.319469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb64f02_7b87_4004_9126_347a2a912e2f.slice/crio-d92d8d7c89c4715a64964c6c6798d0c9dba0b64697ad1aa91e8e3748acb20dab WatchSource:0}: Error finding container d92d8d7c89c4715a64964c6c6798d0c9dba0b64697ad1aa91e8e3748acb20dab: Status 404 returned error can't find the container with id d92d8d7c89c4715a64964c6c6798d0c9dba0b64697ad1aa91e8e3748acb20dab Jul 16 20:33:23.473224 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.473127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:23.473400 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.473278 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:23.473400 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.473360 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:24.473343193 +0000 UTC m=+2.974423903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:23.574542 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.574504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:23.574733 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.574687 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:23.574808 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.574770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:24.574746947 +0000 UTC m=+3.075827668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:23.675616 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.675577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:23.675822 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.675803 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:23.675889 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.675835 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:23.675889 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.675849 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:23.675993 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:23.675912 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:24.675892449 +0000 UTC m=+3.176973172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:23.908551 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.908379 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:28:23 +0000 UTC" deadline="2028-05-03 17:39:12.370558864 +0000 UTC" Jul 16 20:33:23.908551 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:23.908426 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15765h5m48.462138544s" Jul 16 20:33:24.061660 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.061592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"d92d8d7c89c4715a64964c6c6798d0c9dba0b64697ad1aa91e8e3748acb20dab"} Jul 16 20:33:24.068335 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.068127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wn8jn" event={"ID":"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d","Type":"ContainerStarted","Data":"7cfc46ce13bc509601c265d59ca14eb8931830bd048c02844b3d53bef4fa1b5a"} Jul 16 20:33:24.078326 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.078286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h56nk" event={"ID":"dfee882d-4604-4615-b1dc-c7af28f1967f","Type":"ContainerStarted","Data":"f6c1cedb1c093c6e25284bd0787eb7f5038e44fbdc71373047da2639f39e31bf"} Jul 16 20:33:24.086346 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.086298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" event={"ID":"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0","Type":"ContainerStarted","Data":"81f304a21fdc91c45b38b7d32da46285e1d0b19b00b255929e8b91f95cbeaceb"} Jul 16 20:33:24.098828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.098786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f7sgr" event={"ID":"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb","Type":"ContainerStarted","Data":"32bd5fb3f4f567908710f50cf6e425150a46e6d6e7b9eeda4bf30ff36fb628cc"} Jul 16 20:33:24.104174 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.104136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerStarted","Data":"a939c1135e670e6dfa4ca239f38befd0807ccff705805f312a9d82c0c7b34afc"} Jul 16 20:33:24.107549 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.107520 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:24.111646 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.111608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrzzd" event={"ID":"e6d1141b-91ef-4a48-94a9-d33108ccecee","Type":"ContainerStarted","Data":"9f7bd7d1d2401a77387fe80c5e99562ad444347774fbf7651b4291c9b5aef4f2"} Jul 16 20:33:24.130040 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.129998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zjhb" event={"ID":"ee1d15ed-9d78-450f-9a27-ebd324a0394c","Type":"ContainerStarted","Data":"77866b973b15ce85d4ce0a6f777f9d1edb30b233346176bcce529facc7ab5a38"} Jul 16 20:33:24.140162 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.140112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfs5x" event={"ID":"57063013-98eb-4fb8-a531-c9053a008579","Type":"ContainerStarted","Data":"4fe2b607b83a5147844e6cb6a45b608456023479032dfcbb17ecb7f757ddb057"} Jul 16 20:33:24.156362 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.156320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"2d0a72e3dbfc85402677ef718ad352e5","Type":"ContainerStarted","Data":"e6db0fb235bc0d3b5f318ffb35f75b0bdd8b58cd543156a692ec98964c3c54a7"} Jul 16 20:33:24.189502 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.189421 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:24.481713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.481620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:24.481891 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.481803 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:24.481891 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.481869 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.481850672 +0000 UTC m=+4.982931394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:24.582079 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.582028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:24.582273 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.582204 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:24.582358 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.582287 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.582266666 +0000 UTC m=+5.083347387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:24.682989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.682934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:24.683186 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.683153 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:24.683186 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.683178 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:24.683310 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.683191 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:24.683310 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:24.683254 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.683234537 +0000 UTC m=+5.184315246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:24.912640 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.912521 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:28:23 +0000 UTC" deadline="2028-03-09 22:10:04.896375536 +0000 UTC" Jul 16 20:33:24.912640 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:24.912566 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14449h36m39.983814819s" Jul 16 20:33:25.033738 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:25.033702 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:25.033911 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:25.033857 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:25.034404 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:25.034333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:25.034527 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:25.034483 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:25.034591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:25.034561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:25.034894 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:25.034648 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:25.177879 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:25.177627 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:25.442833 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:25.442746 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 20:33:26.502411 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:26.501685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:26.502411 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.501896 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:26.502411 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.501965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:30.501945245 +0000 UTC m=+9.003025954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:26.602393 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:26.602312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:26.602592 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.602557 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:26.602656 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.602626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:30.602605223 +0000 UTC m=+9.103685935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:26.703540 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:26.703497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:26.703716 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.703683 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:26.703716 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.703707 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:26.703849 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.703721 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:26.703849 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:26.703784 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:30.703763794 +0000 UTC m=+9.204844504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:27.033480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:27.033624 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:27.033671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:27.033707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:27.033797 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:27.034000 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:27.033915 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:29.033365 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:29.033332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:29.033786 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:29.033474 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:29.033879 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:29.033854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:29.033986 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:29.033967 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:29.034050 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:29.034015 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:29.034101 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:29.034069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:30.536434 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:30.536198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:30.536921 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.536391 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:30.536921 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.536565 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:38.536538972 +0000 UTC m=+17.037619676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:30.637544 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:30.637500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:30.637725 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.637654 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:30.637725 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.637715 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:38.637695363 +0000 UTC m=+17.138776075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:30.738621 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:30.738548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:30.738820 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.738696 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:30.738820 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.738728 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:30.738820 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.738743 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:30.738982 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:30.738832 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:38.738814927 +0000 UTC m=+17.239895632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:31.033781 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:31.033687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:31.033781 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:31.033757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:31.034013 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:31.033776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:31.034013 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:31.033883 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:31.034013 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:31.033974 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:31.034168 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:31.034067 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:33.033622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:33.033650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:33.033650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:33.033759 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:33.033909 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:33.034129 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:33.033996 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:35.033182 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:35.033151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:35.033630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:35.033151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:35.033630 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:35.033265 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:35.033630 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:35.033151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:35.033630 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:35.033368 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:35.033630 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:35.033436 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:37.033839 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:37.033802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:37.034241 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:37.033805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:37.034241 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:37.033939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:37.034241 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:37.034000 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:37.034241 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:37.033902 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:37.034241 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:37.034082 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:38.596055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:38.596014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:38.596581 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.596142 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:38.596581 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.596207 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:54.596189535 +0000 UTC m=+33.097270254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:38.697294 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:38.697259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:38.697496 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.697411 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:38.697496 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.697473 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:54.697455747 +0000 UTC m=+33.198536457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:38.798152 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:38.798115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:38.798332 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.798280 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:38.798332 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.798322 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:38.798421 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.798336 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:38.798421 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:38.798397 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:54.798378323 +0000 UTC m=+33.299459043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:39.033475 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:39.033433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:39.033647 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:39.033442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:39.033647 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:39.033569 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:39.033767 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:39.033667 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:39.033767 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:39.033447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:39.033857 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:39.033763 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:41.033538 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:41.033495 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:41.034012 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:41.033555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:41.034012 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:41.033555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:41.034012 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:41.033629 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:41.034012 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:41.033727 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:41.034012 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:41.033821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:43.033012 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:43.032975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:43.033484 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:43.032975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:43.033484 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:43.033126 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:43.033484 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:43.032975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:43.033484 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:43.033217 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:43.033484 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:43.033268 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:44.214769 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:33:44.214742 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cc710395b7929d32dc5b7fe48ac3bf.slice/crio-c3cf46adf44e29445ea24e23e3c3d75d08bc44de88e88d4033d61391fcbf1720 WatchSource:0}: Error finding container c3cf46adf44e29445ea24e23e3c3d75d08bc44de88e88d4033d61391fcbf1720: Status 404 returned error can't find the container with id c3cf46adf44e29445ea24e23e3c3d75d08bc44de88e88d4033d61391fcbf1720 Jul 16 20:33:44.221863 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:44.221833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"a3cc710395b7929d32dc5b7fe48ac3bf","Type":"ContainerStarted","Data":"c3cf46adf44e29445ea24e23e3c3d75d08bc44de88e88d4033d61391fcbf1720"} Jul 16 20:33:45.034230 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.034020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:45.034447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.034029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:45.034447 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:45.034336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:45.034447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.034093 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:45.034566 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:45.034506 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:45.034566 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:45.034387 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:45.227589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.227345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" event={"ID":"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0","Type":"ContainerStarted","Data":"ab6c409a5b00be10c793c54623b8c8af1cada0bd53bcefa93add8335ec908f75"} Jul 16 20:33:45.229030 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.228998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f7sgr" event={"ID":"ddf4c0ed-54c3-4ad0-b57f-57b227239cdb","Type":"ContainerStarted","Data":"3c0ba69a18bddad14ea259293e8b4ae147c27fbea998810621cfe89d00d621a6"} Jul 16 20:33:45.231454 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.231427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerStarted","Data":"fb4a85daa43a3857c7f58b8d72c3d1421243112182dd2532805e70a495cfe9b3"} Jul 16 20:33:45.234045 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.234010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrzzd" event={"ID":"e6d1141b-91ef-4a48-94a9-d33108ccecee","Type":"ContainerStarted","Data":"b119f97d00a2d3f28711372d0ee3360086fbf82e12661e2a2646aaf0b79e3d92"} Jul 16 20:33:45.236322 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.236278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zjhb" event={"ID":"ee1d15ed-9d78-450f-9a27-ebd324a0394c","Type":"ContainerStarted","Data":"c3906af5acc9d3bfa1a669adb1bc36356b4702a69cfab04d645eab2d5aedba16"} Jul 16 20:33:45.238775 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.238753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfs5x" event={"ID":"57063013-98eb-4fb8-a531-c9053a008579","Type":"ContainerStarted","Data":"800f6be0cc02b3fe35167af1597c8d0eb834e39d727602daf1d9e7891f8f421f"} Jul 16 20:33:45.240450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.240426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"2d0a72e3dbfc85402677ef718ad352e5","Type":"ContainerStarted","Data":"a460e82e4a84df2e0929b753ed4f817fdb476a8c17e43739cf1871533d71afe8"} Jul 16 20:33:45.242494 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.242478 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:45.242831 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.242810 2564 generic.go:356] "Generic (PLEG): container finished" podID="2cb64f02-7b87-4004-9126-347a2a912e2f" containerID="1c6fdce1dd68253eb170b6bf8c3d947945aaa08438aab713b7c8c950e6ba4848" exitCode=1 Jul 16 20:33:45.242908 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.242891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerDied","Data":"1c6fdce1dd68253eb170b6bf8c3d947945aaa08438aab713b7c8c950e6ba4848"} Jul 16 20:33:45.242974 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.242960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"88647d62e8d795fffb6da7c77f9723f91eebcdfa3f634edb0e1166d1ccfa47de"} Jul 16 20:33:45.244664 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.244636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h56nk" event={"ID":"dfee882d-4604-4615-b1dc-c7af28f1967f","Type":"ContainerStarted","Data":"fc32b084e8caaac40cedbbae16544409f27b9123ad902db7927f4be2167003e0"} Jul 16 20:33:45.245250 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.245204 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f7sgr" podStartSLOduration=2.325369497 podStartE2EDuration="23.245189702s" totalImagesPullingTime="20.919820205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.245051262 +0000 UTC m=+23.746131994" watchObservedRunningTime="2026-07-16 20:33:45.245189702 +0000 UTC m=+23.746270430" Jul 16 20:33:45.257713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.257668 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tfs5x" podStartSLOduration=2.28730833 podStartE2EDuration="23.257655179s" totalImagesPullingTime="20.970346849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.25732985 +0000 UTC m=+23.758410575" watchObservedRunningTime="2026-07-16 20:33:45.257655179 +0000 UTC m=+23.758735904" Jul 16 20:33:45.272468 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.272418 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4zjhb" podStartSLOduration=2.289817656 podStartE2EDuration="23.272399954s" totalImagesPullingTime="20.982582298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.271670216 +0000 UTC m=+23.772750942" watchObservedRunningTime="2026-07-16 20:33:45.272399954 +0000 UTC m=+23.773480677" Jul 16 20:33:45.287648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.287567 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h56nk" podStartSLOduration=2.384617661 podStartE2EDuration="23.287554903s" totalImagesPullingTime="20.902937242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.287223938 +0000 UTC m=+23.788304675" watchObservedRunningTime="2026-07-16 20:33:45.287554903 +0000 UTC m=+23.788635633" Jul 16 20:33:45.320784 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.320734 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" podStartSLOduration=22.320719119 podStartE2EDuration="22.320719119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.320561828 +0000 UTC m=+23.821642556" watchObservedRunningTime="2026-07-16 20:33:45.320719119 +0000 UTC m=+23.821799844" Jul 16 20:33:45.335763 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:45.335716 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mrzzd" podStartSLOduration=2.359252458 podStartE2EDuration="23.335702169s" totalImagesPullingTime="20.976449711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:45.335246229 +0000 UTC m=+23.836326957" watchObservedRunningTime="2026-07-16 20:33:45.335702169 +0000 UTC m=+23.836782894" Jul 16 20:33:46.249877 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.249850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wn8jn" event={"ID":"5066c3d4-7b1d-4fec-a9ba-2fe515879e2d","Type":"ContainerStarted","Data":"cf136a444637b28030f7c80c3714a20057665fcc439036f0073792e3eaa57fe8"} Jul 16 20:33:46.251481 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.251455 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="fb4a85daa43a3857c7f58b8d72c3d1421243112182dd2532805e70a495cfe9b3" exitCode=0 Jul 16 20:33:46.251598 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.251533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"fb4a85daa43a3857c7f58b8d72c3d1421243112182dd2532805e70a495cfe9b3"} Jul 16 20:33:46.253200 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.253176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"a3cc710395b7929d32dc5b7fe48ac3bf","Type":"ContainerStarted","Data":"5162ec2999717245430fc0bfaef9f522c74f6e8c04df5c8f4e7a319ab68069a1"} Jul 16 20:33:46.255253 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.255235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:46.255950 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:46.255600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"134fc853f39329bbce460aed4df0f0a9bf8b19241ff704cf1b23a58b14668d86"} Jul 16 20:33:47.033651 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.033617 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:47.033839 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:47.033751 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:47.033839 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.033801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:47.033939 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:47.033917 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:47.033983 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.033943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:47.034042 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:47.034022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:47.108747 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.108640 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 20:33:47.261470 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.261374 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:47.262055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.261794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"f91945a933f229d5768ae7643a1dd54b92ba28eccf675090e33cb4d49af7a6cf"} Jul 16 20:33:47.262055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.261874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"cab2a18167ce9e0f1f75ea5273a1578e444fcd33d1920913d6c23a1189d6cfd4"} Jul 16 20:33:47.262055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.261940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"35466481fafd893dd57298ccf3e14cd9a0b1d9a1ba831171b5312867f42dc3bf"} Jul 16 20:33:47.264078 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.264046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" event={"ID":"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0","Type":"ContainerStarted","Data":"275294f83872782d5edfc9af053ab60a3d27fdf0c223be4ed240c718a23ba8c3"} Jul 16 20:33:47.265694 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.265666 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3cc710395b7929d32dc5b7fe48ac3bf" containerID="5162ec2999717245430fc0bfaef9f522c74f6e8c04df5c8f4e7a319ab68069a1" exitCode=0 Jul 16 20:33:47.265803 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.265748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"a3cc710395b7929d32dc5b7fe48ac3bf","Type":"ContainerDied","Data":"5162ec2999717245430fc0bfaef9f522c74f6e8c04df5c8f4e7a319ab68069a1"} Jul 16 20:33:47.282467 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.282429 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wn8jn" podStartSLOduration=4.416597725 podStartE2EDuration="25.282413111s" totalImagesPullingTime="20.865815386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:46.27168222 +0000 UTC m=+24.772762946" watchObservedRunningTime="2026-07-16 20:33:47.282413111 +0000 UTC m=+25.783493836" Jul 16 20:33:47.431703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.431668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:47.432495 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.432476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:47.933364 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.933229 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T20:33:47.108656802Z","UUID":"8b33cf7c-e20c-4408-b040-5c400609a907","Handler":null,"Name":"","Endpoint":""} Jul 16 20:33:47.935187 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.935156 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 20:33:47.935343 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:47.935197 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 20:33:48.269929 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.269891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"a3cc710395b7929d32dc5b7fe48ac3bf","Type":"ContainerStarted","Data":"8f18b5e8a297e696817ae477c3f9418b5b163ae81ec075ed6c07875a2f6f34a3"} Jul 16 20:33:48.270405 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.270227 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:48.270754 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.270735 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mrzzd" Jul 16 20:33:48.287841 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.287786 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" podStartSLOduration=25.287768678 podStartE2EDuration="25.287768678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:48.286897135 +0000 UTC m=+26.787977862" watchObservedRunningTime="2026-07-16 20:33:48.287768678 +0000 UTC m=+26.788849406" Jul 16 20:33:48.374412 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.374377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:48.461028 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:48.460990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:49.033580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:49.033543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:49.033757 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:49.033548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:49.033757 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:49.033665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:49.033757 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:49.033677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:49.033924 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:49.033784 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:49.033924 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:49.033884 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:50.277446 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.277186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:50.277986 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.277826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"f5da43972c02b75009c0cf38ee01039b1b2314c3422effeda1e3a99d6e1abcc2"} Jul 16 20:33:50.279787 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.279761 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="5908f42841dd94169281a7a8afcdbc0f173c993bd0ee97e3e0541b3e8a3abfad" exitCode=0 Jul 16 20:33:50.279910 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.279848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"5908f42841dd94169281a7a8afcdbc0f173c993bd0ee97e3e0541b3e8a3abfad"} Jul 16 20:33:50.644845 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.644770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:50.699346 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:50.699316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:51.005387 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.005356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:51.033368 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.033334 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:51.033532 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.033335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:51.033532 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:51.033463 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:51.033532 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.033339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:51.033532 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:51.033500 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:51.033673 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:51.033601 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:51.106733 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.106705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:51.284226 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.284190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" event={"ID":"b56b6b6a-a490-4f8c-a5c6-c0079eea3bd0","Type":"ContainerStarted","Data":"018b90159f8e276cea6b52af595a77bbb20b6562f0a6533a41facabaaa227a3e"} Jul 16 20:33:51.413176 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.413144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:51.471067 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:51.471038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:52.290870 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.290628 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:52.291605 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.291163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"4eb7f396d667b0a1367e505fcdb6999148f0ad8fac3bd90b9f670dda1928fb93"} Jul 16 20:33:52.291605 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.291391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:52.291722 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.291652 2564 scope.go:117] "RemoveContainer" containerID="1c6fdce1dd68253eb170b6bf8c3d947945aaa08438aab713b7c8c950e6ba4848" Jul 16 20:33:52.293578 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.293549 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="d88b5bb585d30ac7ac3dbc78b5bd0c05c2f908cb51bf92b4a3b441f4285d2fcb" exitCode=0 Jul 16 20:33:52.293688 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.293587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"d88b5bb585d30ac7ac3dbc78b5bd0c05c2f908cb51bf92b4a3b441f4285d2fcb"} Jul 16 20:33:52.309105 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.309082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:52.315648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.315607 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzk6h" podStartSLOduration=3.9473547719999997 podStartE2EDuration="30.315593269s" totalImagesPullingTime="26.368238497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:51.310002425 +0000 UTC m=+29.811083141" watchObservedRunningTime="2026-07-16 20:33:52.315593269 +0000 UTC m=+30.816674038" Jul 16 20:33:52.912045 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.912012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:52.988095 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:52.988070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:53.033046 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.033011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:53.033184 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.033108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:53.033184 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:53.033114 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:53.033184 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.033138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:53.033342 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:53.033200 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:53.033342 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:53.033263 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:53.300870 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.300834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:33:53.301296 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.301273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" event={"ID":"2cb64f02-7b87-4004-9126-347a2a912e2f","Type":"ContainerStarted","Data":"880cb8950e93e02a18f323f6fb01ae7ea518ec5ec5655d0d210958766ec43576"} Jul 16 20:33:53.301693 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.301664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:53.301829 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.301815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:53.317760 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.317730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:33:53.335498 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.335437 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" podStartSLOduration=8.757832836 podStartE2EDuration="31.335422378s" totalImagesPullingTime="22.577589542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:53.334866072 +0000 UTC m=+31.835946800" watchObservedRunningTime="2026-07-16 20:33:53.335422378 +0000 UTC m=+31.836503103" Jul 16 20:33:53.468279 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:53.468237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:54.355587 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.355551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jldkm"] Jul 16 20:33:54.355993 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.355709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:54.355993 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.355818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:54.356290 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.356263 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vz888"] Jul 16 20:33:54.356453 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.356372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:54.356510 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.356456 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:54.357141 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.357117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5dddd"] Jul 16 20:33:54.357235 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.357215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:54.357361 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.357341 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:54.607991 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.607947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:54.608198 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.608095 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:54.608198 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.608161 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret podName:157e505d-0be0-4b00-88aa-2d9ab9a68b15 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:26.608145707 +0000 UTC m=+65.109226411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret") pod "global-pull-secret-syncer-vz888" (UID: "157e505d-0be0-4b00-88aa-2d9ab9a68b15") : object "kube-system"/"original-pull-secret" not registered Jul 16 20:33:54.708610 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.708568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:54.708757 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.708711 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:54.708812 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.708782 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs podName:23fc2b71-d828-4b25-9122-c09e1bb02140 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:26.708762678 +0000 UTC m=+65.209843385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs") pod "network-metrics-daemon-5dddd" (UID: "23fc2b71-d828-4b25-9122-c09e1bb02140") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 20:33:54.810001 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:54.809968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:54.810193 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.810172 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 20:33:54.810234 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.810202 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 20:33:54.810234 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.810216 2564 projected.go:196] Error preparing data for projected volume kube-api-access-4j4sg for pod openshift-network-diagnostics/network-check-target-jldkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:54.810344 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:54.810285 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg podName:b26e994e-cb1b-40d2-88ba-035cdf6b232e nodeName:}" failed. No retries permitted until 2026-07-16 20:34:26.810262008 +0000 UTC m=+65.311342736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j4sg" (UniqueName: "kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg") pod "network-check-target-jldkm" (UID: "b26e994e-cb1b-40d2-88ba-035cdf6b232e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 20:33:56.034042 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:56.033855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:56.034466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:56.033870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:56.034466 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:56.034128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:56.034466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:56.033881 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:56.034466 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:56.034222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:56.034466 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:56.034272 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:56.313214 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:56.313129 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="0d183eecfcdd69456d55b6bc2e0597180f6d706b6f32a13577d08c58c411ec3e" exitCode=0 Jul 16 20:33:56.313214 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:56.313170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"0d183eecfcdd69456d55b6bc2e0597180f6d706b6f32a13577d08c58c411ec3e"} Jul 16 20:33:57.198635 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.198584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:57.265694 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.265655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:57.428615 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.428579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:57.534011 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.533979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:57.829422 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.829383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:57.906903 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:57.906870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.033434 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.033396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:33:58.033620 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.033396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:33:58.033620 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.033529 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jldkm" podUID="b26e994e-cb1b-40d2-88ba-035cdf6b232e" Jul 16 20:33:58.033620 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.033571 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz888" podUID="157e505d-0be0-4b00-88aa-2d9ab9a68b15" Jul 16 20:33:58.033620 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.033396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:33:58.033758 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.033657 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5dddd" podUID="23fc2b71-d828-4b25-9122-c09e1bb02140" Jul 16 20:33:58.059028 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.058991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.162596 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.162517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.286639 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.286590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.384646 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.384613 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeReady" Jul 16 20:33:58.384820 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.384771 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 20:33:58.386039 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.385979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.450662 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.450580 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2xs5q"] Jul 16 20:33:58.452372 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.452351 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kjhm4"] Jul 16 20:33:58.452497 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.452464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.454190 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.454171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.455110 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.455096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 20:33:58.455445 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.455427 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 20:33:58.455500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.455456 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 20:33:58.455976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.455963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c96n9\"" Jul 16 20:33:58.456721 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.456705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 20:33:58.456813 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.456761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q8qkr\"" Jul 16 20:33:58.456878 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.456826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 20:33:58.466365 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.466342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjhm4"] Jul 16 20:33:58.467075 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.467044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2xs5q"] Jul 16 20:33:58.474894 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.474872 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8rfz7"] Jul 16 20:33:58.476936 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.476923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.479331 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.479289 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 20:33:58.479532 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.479517 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 20:33:58.479580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.479558 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 20:33:58.479755 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.479741 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ljszm\"" Jul 16 20:33:58.479799 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.479760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 20:33:58.486789 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.486768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.488036 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.488018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rfz7"] Jul 16 20:33:58.539111 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpwq\" (UniqueName: \"kubernetes.io/projected/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-kube-api-access-mtpwq\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.539111 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.539359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-tmp-dir\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.539359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37db4d2c-a03f-43be-bac1-a659cb4211b0-cert\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.539359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdq6\" (UniqueName: \"kubernetes.io/projected/37db4d2c-a03f-43be-bac1-a659cb4211b0-kube-api-access-6rdq6\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.539359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71869283-694c-4d79-8fc0-e38a3e65c3e7-data-volume\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.539359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.539569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjd6\" (UniqueName: \"kubernetes.io/projected/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-api-access-9rjd6\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.539569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-config-volume\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.539569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.539569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.539476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71869283-694c-4d79-8fc0-e38a3e65c3e7-crio-socket\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.568363 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.568320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:33:58.640625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjd6\" (UniqueName: \"kubernetes.io/projected/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-api-access-9rjd6\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.640625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-config-volume\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71869283-694c-4d79-8fc0-e38a3e65c3e7-crio-socket\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpwq\" (UniqueName: \"kubernetes.io/projected/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-kube-api-access-mtpwq\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-tmp-dir\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.640783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37db4d2c-a03f-43be-bac1-a659cb4211b0-cert\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.640781 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 20:33:58.640846 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.640849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls podName:86d018bd-ec4f-4ac6-9dd7-34a0c2087f81 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:59.140830861 +0000 UTC m=+37.641911579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls") pod "dns-default-kjhm4" (UID: "86d018bd-ec4f-4ac6-9dd7-34a0c2087f81") : secret "dns-default-metrics-tls" not found Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.640877 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.640944 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:59.140926918 +0000 UTC m=+37.642007642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : configmap "kube-rbac-proxy" not found Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-tmp-dir\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdq6\" (UniqueName: \"kubernetes.io/projected/37db4d2c-a03f-43be-bac1-a659cb4211b0-kube-api-access-6rdq6\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71869283-694c-4d79-8fc0-e38a3e65c3e7-crio-socket\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.641233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71869283-694c-4d79-8fc0-e38a3e65c3e7-data-volume\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.641511 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.641511 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-config-volume\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.641511 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.641397 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 20:33:58.641511 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:58.641446 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:59.141434054 +0000 UTC m=+37.642514759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : secret "insights-runtime-extractor-tls" not found Jul 16 20:33:58.641693 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.641596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71869283-694c-4d79-8fc0-e38a3e65c3e7-data-volume\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.650589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.650561 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpwq\" (UniqueName: \"kubernetes.io/projected/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-kube-api-access-mtpwq\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:58.650746 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.650647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjd6\" (UniqueName: \"kubernetes.io/projected/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-api-access-9rjd6\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:58.650996 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.650974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37db4d2c-a03f-43be-bac1-a659cb4211b0-cert\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.656409 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.656384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdq6\" (UniqueName: \"kubernetes.io/projected/37db4d2c-a03f-43be-bac1-a659cb4211b0-kube-api-access-6rdq6\") pod \"ingress-canary-2xs5q\" (UID: \"37db4d2c-a03f-43be-bac1-a659cb4211b0\") " pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.743978 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.743950 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f7sgr_ddf4c0ed-54c3-4ad0-b57f-57b227239cdb/dns-node-resolver/0.log" Jul 16 20:33:58.763037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.762982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2xs5q" Jul 16 20:33:58.921925 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:58.921756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2xs5q"] Jul 16 20:33:59.144394 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:59.144256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:59.144394 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:59.144352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144409 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144474 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:00.14445673 +0000 UTC m=+38.645537434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : configmap "kube-rbac-proxy" not found Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:33:59.144416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144505 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144510 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 20:33:59.144624 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls podName:86d018bd-ec4f-4ac6-9dd7-34a0c2087f81 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:00.144604871 +0000 UTC m=+38.645685585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls") pod "dns-default-kjhm4" (UID: "86d018bd-ec4f-4ac6-9dd7-34a0c2087f81") : secret "dns-default-metrics-tls" not found Jul 16 20:33:59.144862 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:33:59.144673 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:00.14465397 +0000 UTC m=+38.645734689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : secret "insights-runtime-extractor-tls" not found Jul 16 20:34:00.033943 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.033903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:34:00.034743 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.033904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:34:00.034743 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.034087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:34:00.037956 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.037929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss679\"" Jul 16 20:34:00.038157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.037966 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 20:34:00.038157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.037977 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 20:34:00.038157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.037994 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xxdx\"" Jul 16 20:34:00.038157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.037942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 20:34:00.038157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.038069 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 20:34:00.131005 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.130973 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8l28p"] Jul 16 20:34:00.133541 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.133521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.136230 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.136182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 20:34:00.136423 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.136403 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q78h4\"" Jul 16 20:34:00.136530 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.136499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 20:34:00.136589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.136535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 20:34:00.136763 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.136746 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 20:34:00.139579 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.139560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8l28p"] Jul 16 20:34:00.144214 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.144189 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfs5x_57063013-98eb-4fb8-a531-c9053a008579/node-ca/0.log" Jul 16 20:34:00.152828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.152795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:00.152949 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.152890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:00.152949 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.152931 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 20:34:00.152949 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.152944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:00.153100 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.153004 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:02.152987641 +0000 UTC m=+40.654068367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : secret "insights-runtime-extractor-tls" not found Jul 16 20:34:00.153100 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.153039 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 20:34:00.153100 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.153041 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 20:34:00.153100 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.153089 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:02.153071739 +0000 UTC m=+40.654152445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : configmap "kube-rbac-proxy" not found Jul 16 20:34:00.153286 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:00.153110 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls podName:86d018bd-ec4f-4ac6-9dd7-34a0c2087f81 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:02.153095114 +0000 UTC m=+40.654175823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls") pod "dns-default-kjhm4" (UID: "86d018bd-ec4f-4ac6-9dd7-34a0c2087f81") : secret "dns-default-metrics-tls" not found Jul 16 20:34:00.254050 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.254013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-key\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.254269 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.254248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-cabundle\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.254362 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.254287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n62\" (UniqueName: \"kubernetes.io/projected/af92eeb8-173e-41f3-bdbc-2d3b71396c47-kube-api-access-r2n62\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.354801 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.354715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-cabundle\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.354801 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.354764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n62\" (UniqueName: \"kubernetes.io/projected/af92eeb8-173e-41f3-bdbc-2d3b71396c47-kube-api-access-r2n62\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.355020 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.354909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-key\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.355540 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.355515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-cabundle\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.358902 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.358872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af92eeb8-173e-41f3-bdbc-2d3b71396c47-signing-key\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.363807 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.363689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n62\" (UniqueName: \"kubernetes.io/projected/af92eeb8-173e-41f3-bdbc-2d3b71396c47-kube-api-access-r2n62\") pod \"service-ca-58598695f7-8l28p\" (UID: \"af92eeb8-173e-41f3-bdbc-2d3b71396c47\") " pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.445633 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.445600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-8l28p" Jul 16 20:34:00.445917 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:00.445893 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37db4d2c_a03f_43be_bac1_a659cb4211b0.slice/crio-d981b0acc8398ab117d8629def7af483764c730b165310c4dfc9f141b9b81ea2 WatchSource:0}: Error finding container d981b0acc8398ab117d8629def7af483764c730b165310c4dfc9f141b9b81ea2: Status 404 returned error can't find the container with id d981b0acc8398ab117d8629def7af483764c730b165310c4dfc9f141b9b81ea2 Jul 16 20:34:00.584031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:00.584007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-8l28p"] Jul 16 20:34:00.588022 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:00.587991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf92eeb8_173e_41f3_bdbc_2d3b71396c47.slice/crio-48a3dbca3fdb8070d2f260bdeecb5b124675dbc0fc1b163a5225958d92254401 WatchSource:0}: Error finding container 48a3dbca3fdb8070d2f260bdeecb5b124675dbc0fc1b163a5225958d92254401: Status 404 returned error can't find the container with id 48a3dbca3fdb8070d2f260bdeecb5b124675dbc0fc1b163a5225958d92254401 Jul 16 20:34:01.330860 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:01.330821 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="6497ebae1e03e79464b65b7b0bcf35f0e89eb8043ce7a38b992cdc962ec56e15" exitCode=0 Jul 16 20:34:01.331344 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:01.330924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"6497ebae1e03e79464b65b7b0bcf35f0e89eb8043ce7a38b992cdc962ec56e15"} Jul 16 20:34:01.332506 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:01.332475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8l28p" event={"ID":"af92eeb8-173e-41f3-bdbc-2d3b71396c47","Type":"ContainerStarted","Data":"48a3dbca3fdb8070d2f260bdeecb5b124675dbc0fc1b163a5225958d92254401"} Jul 16 20:34:01.333779 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:01.333735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2xs5q" event={"ID":"37db4d2c-a03f-43be-bac1-a659cb4211b0","Type":"ContainerStarted","Data":"d981b0acc8398ab117d8629def7af483764c730b165310c4dfc9f141b9b81ea2"} Jul 16 20:34:02.159377 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.159139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:02.170556 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.170512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:02.170720 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.170592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:02.170720 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.170625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:02.170720 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170654 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 20:34:02.170720 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170715 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 20:34:02.170871 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170728 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:06.170706879 +0000 UTC m=+44.671787606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : secret "insights-runtime-extractor-tls" not found Jul 16 20:34:02.170871 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170741 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 20:34:02.170871 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170765 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:06.170750557 +0000 UTC m=+44.671831282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : configmap "kube-rbac-proxy" not found Jul 16 20:34:02.170871 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:02.170783 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls podName:86d018bd-ec4f-4ac6-9dd7-34a0c2087f81 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:06.170772397 +0000 UTC m=+44.671853130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls") pod "dns-default-kjhm4" (UID: "86d018bd-ec4f-4ac6-9dd7-34a0c2087f81") : secret "dns-default-metrics-tls" not found Jul 16 20:34:02.260081 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.260044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:02.340076 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.340044 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7d430ca-5628-4814-9959-e63438410d86" containerID="cf4854aeba8a7d36dcb39889b11c27604ce94aeabff21dd568bede69b7268b81" exitCode=0 Jul 16 20:34:02.340510 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:02.340115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerDied","Data":"cf4854aeba8a7d36dcb39889b11c27604ce94aeabff21dd568bede69b7268b81"} Jul 16 20:34:03.513865 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:03.513828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.292218 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.292182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.347914 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.347876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" event={"ID":"c7d430ca-5628-4814-9959-e63438410d86","Type":"ContainerStarted","Data":"ac7163d3fc7523a4a4710dee2386e1ac5dbf997e87fccf52d070e1dba0e22b4e"} Jul 16 20:34:04.349379 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.349352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-8l28p" event={"ID":"af92eeb8-173e-41f3-bdbc-2d3b71396c47","Type":"ContainerStarted","Data":"c492e4fc7cfccf2f818a186f7c3b6855cf87113b562745389a9e4892078cc8fd"} Jul 16 20:34:04.350725 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.350692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2xs5q" event={"ID":"37db4d2c-a03f-43be-bac1-a659cb4211b0","Type":"ContainerStarted","Data":"3dc5c9d17d13f6f48b21fa1b524bf771a1a9fb137740ac54438ecb8ed7545760"} Jul 16 20:34:04.374487 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.374441 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s2gq9" podStartSLOduration=7.9175188720000005 podStartE2EDuration="42.374425462s" totalImagesPullingTime="34.45690659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:04.37256655 +0000 UTC m=+42.873647288" watchObservedRunningTime="2026-07-16 20:34:04.374425462 +0000 UTC m=+42.875506210" Jul 16 20:34:04.395911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.395849 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-8l28p" podStartSLOduration=1.318507883 podStartE2EDuration="4.39583087s" totalImagesPullingTime="3.077322987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:04.394970331 +0000 UTC m=+42.896051058" watchObservedRunningTime="2026-07-16 20:34:04.39583087 +0000 UTC m=+42.896911573" Jul 16 20:34:04.411341 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.411270 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2xs5q" podStartSLOduration=3.197089634 podStartE2EDuration="6.411248446s" totalImagesPullingTime="3.214158812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:04.410593772 +0000 UTC m=+42.911674498" watchObservedRunningTime="2026-07-16 20:34:04.411248446 +0000 UTC m=+42.912329175" Jul 16 20:34:04.588360 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.588231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.695514 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.695476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.812417 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.812374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.910988 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.910893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:04.984917 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:04.984878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:05.366130 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:05.366079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:06.071600 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.071571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:06.206127 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.206079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:06.206294 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.206161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:06.206294 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.206202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:06.206294 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:06.206229 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 20:34:06.206294 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:06.206287 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls podName:71869283-694c-4d79-8fc0-e38a3e65c3e7 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:14.206271196 +0000 UTC m=+52.707351900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rfz7" (UID: "71869283-694c-4d79-8fc0-e38a3e65c3e7") : secret "insights-runtime-extractor-tls" not found Jul 16 20:34:06.206560 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:06.206335 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 20:34:06.206560 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:06.206398 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls podName:86d018bd-ec4f-4ac6-9dd7-34a0c2087f81 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:14.206379543 +0000 UTC m=+52.707460253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls") pod "dns-default-kjhm4" (UID: "86d018bd-ec4f-4ac6-9dd7-34a0c2087f81") : secret "dns-default-metrics-tls" not found Jul 16 20:34:06.206842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.206819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71869283-694c-4d79-8fc0-e38a3e65c3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:06.493857 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.493824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:06.869002 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:06.868914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:07.118469 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:07.118424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:07.699645 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:07.699611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.257506 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.257472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.374328 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.374284 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.472199 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.472162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.574117 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.574028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.674791 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.674756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.864997 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.864897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:08.966218 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:08.966175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.069808 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.069778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.124543 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.124466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.468104 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.468020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.678914 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.678881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.876098 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.876056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:09.973703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:09.973663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:10.078686 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:10.078652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:10.180157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:10.180080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:10.473092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:10.473057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:10.777545 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:10.777466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:11.066959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:11.066885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:11.268997 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:11.268969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:11.471893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:11.471858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:11.874754 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:11.874677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:12.664133 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:12.664093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:13.068569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:13.068532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:13.267589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:13.267543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:13.465286 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:13.465196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:13.872311 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:13.872276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:14.091257 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.091211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:14.192684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.192613 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:14.269380 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.269341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:14.269538 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.269440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:14.272769 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.272746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71869283-694c-4d79-8fc0-e38a3e65c3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rfz7\" (UID: \"71869283-694c-4d79-8fc0-e38a3e65c3e7\") " pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:14.272880 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.272816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86d018bd-ec4f-4ac6-9dd7-34a0c2087f81-metrics-tls\") pod \"dns-default-kjhm4\" (UID: \"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81\") " pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:14.346710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.346681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:14.368599 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.368570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:14.385407 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.385380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rfz7" Jul 16 20:34:14.468892 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.468860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:14.554722 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.554692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjhm4"] Jul 16 20:34:14.557567 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:14.557537 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d018bd_ec4f_4ac6_9dd7_34a0c2087f81.slice/crio-02b788d30d1ca9a0b71f3367a93b533c98d4b11b8a666b37912e6f3ff4ffa88f WatchSource:0}: Error finding container 02b788d30d1ca9a0b71f3367a93b533c98d4b11b8a666b37912e6f3ff4ffa88f: Status 404 returned error can't find the container with id 02b788d30d1ca9a0b71f3367a93b533c98d4b11b8a666b37912e6f3ff4ffa88f Jul 16 20:34:14.570496 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:14.570466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rfz7"] Jul 16 20:34:14.573687 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:14.573662 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71869283_694c_4d79_8fc0_e38a3e65c3e7.slice/crio-e914689e779607bcee713c8e0132b47107cb66df137a5819325bd4508c303241 WatchSource:0}: Error finding container e914689e779607bcee713c8e0132b47107cb66df137a5819325bd4508c303241: Status 404 returned error can't find the container with id e914689e779607bcee713c8e0132b47107cb66df137a5819325bd4508c303241 Jul 16 20:34:15.071943 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.071896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:15.272278 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.272242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:15.379036 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.378950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjhm4" event={"ID":"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81","Type":"ContainerStarted","Data":"02b788d30d1ca9a0b71f3367a93b533c98d4b11b8a666b37912e6f3ff4ffa88f"} Jul 16 20:34:15.380776 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.380745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rfz7" event={"ID":"71869283-694c-4d79-8fc0-e38a3e65c3e7","Type":"ContainerStarted","Data":"e914689e779607bcee713c8e0132b47107cb66df137a5819325bd4508c303241"} Jul 16 20:34:15.468186 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.468113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:15.664528 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:15.664400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:17.891038 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:17.890994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:17.945760 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:17.945722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:18.436047 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:18.436003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:18.491288 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:18.491252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:20.397293 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:20.397253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rfz7" event={"ID":"71869283-694c-4d79-8fc0-e38a3e65c3e7","Type":"ContainerStarted","Data":"6be6bdbd3a159a2936f87bf81cb7114ee8e649c5f76262a5b1168e6507b7b0d0"} Jul 16 20:34:20.398953 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:20.398931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjhm4" event={"ID":"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81","Type":"ContainerStarted","Data":"757fd1c46f2a769610184be83c54548fbaf16ec966e6bbc36db681c1a388dd05"} Jul 16 20:34:20.399054 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:20.398970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjhm4" event={"ID":"86d018bd-ec4f-4ac6-9dd7-34a0c2087f81","Type":"ContainerStarted","Data":"231ca6dc07959b47da9ee12ba8afb1eedc7e413fe2bf5f38f5032d846ef81d24"} Jul 16 20:34:20.399092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:20.399057 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:20.417134 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:20.417087 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kjhm4" podStartSLOduration=17.009337617 podStartE2EDuration="22.417072888s" totalImagesPullingTime="5.407735271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:20.415535986 +0000 UTC m=+58.916616715" watchObservedRunningTime="2026-07-16 20:34:20.417072888 +0000 UTC m=+58.918153613" Jul 16 20:34:21.096809 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:21.096779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:21.156926 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:21.156892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:23.132332 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:23.132282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:23.938450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:23.938415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f"] Jul 16 20:34:24.011129 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.011098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f"] Jul 16 20:34:24.011326 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.011231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.014729 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.014707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 20:34:24.015040 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.015013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 20:34:24.015165 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.015025 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 20:34:24.016264 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.016113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 20:34:24.016264 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.016121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 20:34:24.016264 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.016170 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 20:34:24.016593 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.016571 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 20:34:24.026519 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.026491 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657968875d-v8h6h"] Jul 16 20:34:24.031487 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.031459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:24.046784 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.046784 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6eec1d-8d07-49e6-a44a-01f982386a55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.047017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.047017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.047017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfk7\" (UniqueName: \"kubernetes.io/projected/ce6eec1d-8d07-49e6-a44a-01f982386a55-kube-api-access-xdfk7\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.047017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.046898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.053041 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.053017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.056017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.055981 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 20:34:24.056017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.055999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v7sr7\"" Jul 16 20:34:24.056619 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.056598 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 20:34:24.057336 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.057296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657968875d-v8h6h"] Jul 16 20:34:24.057855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.057837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 20:34:24.060661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.060642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 20:34:24.148390 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-trusted-ca\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-certificates\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5502967-22ad-415d-9ca3-d4a20e5893e8-ca-trust-extracted\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-bound-sa-token\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6eec1d-8d07-49e6-a44a-01f982386a55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfk7\" (UniqueName: \"kubernetes.io/projected/ce6eec1d-8d07-49e6-a44a-01f982386a55-kube-api-access-xdfk7\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-tls\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-installation-pull-secrets\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btflp\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-kube-api-access-btflp\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.148815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.148792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-image-registry-private-configuration\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.149769 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.149274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6eec1d-8d07-49e6-a44a-01f982386a55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.151953 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.151917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.152079 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.151965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.152079 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.152043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-ca\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.156967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.156946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce6eec1d-8d07-49e6-a44a-01f982386a55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.159883 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.159858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfk7\" (UniqueName: \"kubernetes.io/projected/ce6eec1d-8d07-49e6-a44a-01f982386a55-kube-api-access-xdfk7\") pod \"cluster-proxy-proxy-agent-5c97957859-wb72f\" (UID: \"ce6eec1d-8d07-49e6-a44a-01f982386a55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.189238 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.189158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:24.249447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-bound-sa-token\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-tls\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-installation-pull-secrets\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btflp\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-kube-api-access-btflp\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-image-registry-private-configuration\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-trusted-ca\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-certificates\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.249700 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.249625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5502967-22ad-415d-9ca3-d4a20e5893e8-ca-trust-extracted\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.250076 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.250041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5502967-22ad-415d-9ca3-d4a20e5893e8-ca-trust-extracted\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.251272 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.251242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-certificates\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.251606 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.251558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5502967-22ad-415d-9ca3-d4a20e5893e8-trusted-ca\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.252989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.252871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-image-registry-private-configuration\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.252989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.252977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5502967-22ad-415d-9ca3-d4a20e5893e8-installation-pull-secrets\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.253142 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.253050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-registry-tls\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.258060 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.258037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-bound-sa-token\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.258180 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.258043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btflp\" (UniqueName: \"kubernetes.io/projected/c5502967-22ad-415d-9ca3-d4a20e5893e8-kube-api-access-btflp\") pod \"image-registry-657968875d-v8h6h\" (UID: \"c5502967-22ad-415d-9ca3-d4a20e5893e8\") " pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:24.290534 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.290452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:24.332467 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.332432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" Jul 16 20:34:24.366474 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:24.366440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:25.323650 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:25.323620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6xlh" Jul 16 20:34:25.444026 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:25.443993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:25.510038 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:25.510001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:25.855343 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:25.855294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657968875d-v8h6h"] Jul 16 20:34:25.858833 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:25.858809 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5502967_22ad_415d_9ca3_d4a20e5893e8.slice/crio-3de971cf3a885fcd4cdcf3f0c2e7075f71261243d7b4d362dc2cd937f6eb6404 WatchSource:0}: Error finding container 3de971cf3a885fcd4cdcf3f0c2e7075f71261243d7b4d362dc2cd937f6eb6404: Status 404 returned error can't find the container with id 3de971cf3a885fcd4cdcf3f0c2e7075f71261243d7b4d362dc2cd937f6eb6404 Jul 16 20:34:25.869575 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:25.869558 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f"] Jul 16 20:34:25.871723 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:25.871701 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6eec1d_8d07_49e6_a44a_01f982386a55.slice/crio-b6f971ceb0593493161f9182a6394e5eeabbab5ce6321c08092e26666368396e WatchSource:0}: Error finding container b6f971ceb0593493161f9182a6394e5eeabbab5ce6321c08092e26666368396e: Status 404 returned error can't find the container with id b6f971ceb0593493161f9182a6394e5eeabbab5ce6321c08092e26666368396e Jul 16 20:34:26.421236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.420896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657968875d-v8h6h" event={"ID":"c5502967-22ad-415d-9ca3-d4a20e5893e8","Type":"ContainerStarted","Data":"a9efc54e2a386b99b6474d80a098e682a1775dd68566f1a4984851c9c4f0f0ef"} Jul 16 20:34:26.421236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.420954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657968875d-v8h6h" event={"ID":"c5502967-22ad-415d-9ca3-d4a20e5893e8","Type":"ContainerStarted","Data":"3de971cf3a885fcd4cdcf3f0c2e7075f71261243d7b4d362dc2cd937f6eb6404"} Jul 16 20:34:26.421236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.420999 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:26.423359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.423328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rfz7" event={"ID":"71869283-694c-4d79-8fc0-e38a3e65c3e7","Type":"ContainerStarted","Data":"b1314c85906597f884ca2339d23e71e7bd0258ecd80544d422237e75ff32cca4"} Jul 16 20:34:26.424722 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.424697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" event={"ID":"ce6eec1d-8d07-49e6-a44a-01f982386a55","Type":"ContainerStarted","Data":"b6f971ceb0593493161f9182a6394e5eeabbab5ce6321c08092e26666368396e"} Jul 16 20:34:26.442341 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.441280 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657968875d-v8h6h" podStartSLOduration=2.441261378 podStartE2EDuration="2.441261378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:26.440747446 +0000 UTC m=+64.941828174" watchObservedRunningTime="2026-07-16 20:34:26.441261378 +0000 UTC m=+64.942342082" Jul 16 20:34:26.669691 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.669633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:34:26.672418 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.672347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 20:34:26.683148 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.683121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/157e505d-0be0-4b00-88aa-2d9ab9a68b15-original-pull-secret\") pod \"global-pull-secret-syncer-vz888\" (UID: \"157e505d-0be0-4b00-88aa-2d9ab9a68b15\") " pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:34:26.754266 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.754229 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz888" Jul 16 20:34:26.770444 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.770413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:34:26.772988 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.772963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 20:34:26.783569 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.783543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23fc2b71-d828-4b25-9122-c09e1bb02140-metrics-certs\") pod \"network-metrics-daemon-5dddd\" (UID: \"23fc2b71-d828-4b25-9122-c09e1bb02140\") " pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:34:26.871787 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.871757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:34:26.874507 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.874472 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 20:34:26.885255 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.885231 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 20:34:26.897619 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.897590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4sg\" (UniqueName: \"kubernetes.io/projected/b26e994e-cb1b-40d2-88ba-035cdf6b232e-kube-api-access-4j4sg\") pod \"network-check-target-jldkm\" (UID: \"b26e994e-cb1b-40d2-88ba-035cdf6b232e\") " pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:34:26.902747 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:26.902715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vz888"] Jul 16 20:34:26.906380 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:26.906349 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157e505d_0be0_4b00_88aa_2d9ab9a68b15.slice/crio-a91ed61a3be353ca0c77ca1267b3b8c83e191c65448af4662bca39a7fc4fed2e WatchSource:0}: Error finding container a91ed61a3be353ca0c77ca1267b3b8c83e191c65448af4662bca39a7fc4fed2e: Status 404 returned error can't find the container with id a91ed61a3be353ca0c77ca1267b3b8c83e191c65448af4662bca39a7fc4fed2e Jul 16 20:34:27.050976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.050866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss679\"" Jul 16 20:34:27.052454 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.052429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:27.059058 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.059036 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5dddd" Jul 16 20:34:27.062857 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.062834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xxdx\"" Jul 16 20:34:27.071127 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.071102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:34:27.108553 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.108517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:27.238069 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.237817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5dddd"] Jul 16 20:34:27.241239 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:27.241181 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fc2b71_d828_4b25_9122_c09e1bb02140.slice/crio-30d4309a4ea241c0e96b24fadbf9aee501cef62a4cbb08a34e6e6a41715e3c6b WatchSource:0}: Error finding container 30d4309a4ea241c0e96b24fadbf9aee501cef62a4cbb08a34e6e6a41715e3c6b: Status 404 returned error can't find the container with id 30d4309a4ea241c0e96b24fadbf9aee501cef62a4cbb08a34e6e6a41715e3c6b Jul 16 20:34:27.254097 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.253947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jldkm"] Jul 16 20:34:27.260465 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:27.260437 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26e994e_cb1b_40d2_88ba_035cdf6b232e.slice/crio-e05b7722ad71a7f36b147d8fc7697ef9313ab24b6a99bd2908bb733a841c26f6 WatchSource:0}: Error finding container e05b7722ad71a7f36b147d8fc7697ef9313ab24b6a99bd2908bb733a841c26f6: Status 404 returned error can't find the container with id e05b7722ad71a7f36b147d8fc7697ef9313ab24b6a99bd2908bb733a841c26f6 Jul 16 20:34:27.344549 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.344460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:27.403601 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.403569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:27.430458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.430415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5dddd" event={"ID":"23fc2b71-d828-4b25-9122-c09e1bb02140","Type":"ContainerStarted","Data":"30d4309a4ea241c0e96b24fadbf9aee501cef62a4cbb08a34e6e6a41715e3c6b"} Jul 16 20:34:27.432055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.432015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jldkm" event={"ID":"b26e994e-cb1b-40d2-88ba-035cdf6b232e","Type":"ContainerStarted","Data":"e05b7722ad71a7f36b147d8fc7697ef9313ab24b6a99bd2908bb733a841c26f6"} Jul 16 20:34:27.433609 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:27.433587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vz888" event={"ID":"157e505d-0be0-4b00-88aa-2d9ab9a68b15","Type":"ContainerStarted","Data":"a91ed61a3be353ca0c77ca1267b3b8c83e191c65448af4662bca39a7fc4fed2e"} Jul 16 20:34:28.088133 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.088093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.174333 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.174247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.257160 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.257099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.333424 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.333359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.515196 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.515159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.571145 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.571113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:28.888370 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:28.888264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.114722 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.114645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.169578 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.169493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.501813 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.501779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.558914 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.558847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.914338 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.914233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:29.975545 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:29.975511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:30.404962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:30.404918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kjhm4" Jul 16 20:34:32.498982 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:32.498938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:33.424732 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:33.424694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:33.481020 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:33.480985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:33.701761 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:33.701671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:36.283211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.283178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:36.350289 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.350257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:36.466333 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.466278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jldkm" event={"ID":"b26e994e-cb1b-40d2-88ba-035cdf6b232e","Type":"ContainerStarted","Data":"ab2b05e9440ca5d321b8b756c1d9f3270f8ea8d4bc52ae186cd74723aacf9a92"} Jul 16 20:34:36.466527 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.466507 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:34:36.467836 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.467814 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vz888" event={"ID":"157e505d-0be0-4b00-88aa-2d9ab9a68b15","Type":"ContainerStarted","Data":"3c6aa18a77ea455a670c141906cc03832c953e65ced1b86eea2cf5ce674175b8"} Jul 16 20:34:36.469993 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.469970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rfz7" event={"ID":"71869283-694c-4d79-8fc0-e38a3e65c3e7","Type":"ContainerStarted","Data":"c8d94adbc4cf94491c1be6a117cbf7bdedc8a8503d81b27f8e3c45c0318e8402"} Jul 16 20:34:36.471992 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.471969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" event={"ID":"ce6eec1d-8d07-49e6-a44a-01f982386a55","Type":"ContainerStarted","Data":"1c48d4bfb7dd8b77670a8636155e5720cc4919d03abb9cb3626af8335bd856ab"} Jul 16 20:34:36.472060 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.472005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" event={"ID":"ce6eec1d-8d07-49e6-a44a-01f982386a55","Type":"ContainerStarted","Data":"1ab0f5e41a880d6b9c0f8aa1221787f83189406eeee5b4990492315148a5e592"} Jul 16 20:34:36.472060 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.472022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" event={"ID":"ce6eec1d-8d07-49e6-a44a-01f982386a55","Type":"ContainerStarted","Data":"d558012c0228c1fa8ec1370c6173203b5d4db66425a5744051d516e2aeca92f8"} Jul 16 20:34:36.473745 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.473723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5dddd" event={"ID":"23fc2b71-d828-4b25-9122-c09e1bb02140","Type":"ContainerStarted","Data":"6a64d5c7c2082f531c07199317554228c4cbfc9a494bee4e68275cb5f6fb7542"} Jul 16 20:34:36.473847 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.473767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5dddd" event={"ID":"23fc2b71-d828-4b25-9122-c09e1bb02140","Type":"ContainerStarted","Data":"c96c231786b7e58df9d3122e0fade3be01434f89991c6a27b9106526d8f8dd9f"} Jul 16 20:34:36.481550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.481507 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jldkm" podStartSLOduration=65.917174712 podStartE2EDuration="1m14.481493196s" totalImagesPullingTime="8.564318484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:36.480911349 +0000 UTC m=+74.981992074" watchObservedRunningTime="2026-07-16 20:34:36.481493196 +0000 UTC m=+74.982573923" Jul 16 20:34:36.498968 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.498922 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8rfz7" podStartSLOduration=17.430004425 podStartE2EDuration="38.498905986s" totalImagesPullingTime="21.068901561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:36.498256187 +0000 UTC m=+74.999336914" watchObservedRunningTime="2026-07-16 20:34:36.498905986 +0000 UTC m=+74.999986712" Jul 16 20:34:36.518265 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.518219 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c97957859-wb72f" podStartSLOduration=3.601958937 podStartE2EDuration="13.51820485s" totalImagesPullingTime="9.916245913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:36.51727429 +0000 UTC m=+75.018355021" watchObservedRunningTime="2026-07-16 20:34:36.51820485 +0000 UTC m=+75.019285577" Jul 16 20:34:36.531665 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.531606 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5dddd" podStartSLOduration=65.985032196 podStartE2EDuration="1m14.531592259s" totalImagesPullingTime="8.546560063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:36.531215636 +0000 UTC m=+75.032296362" watchObservedRunningTime="2026-07-16 20:34:36.531592259 +0000 UTC m=+75.032673032" Jul 16 20:34:36.545199 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:36.545112 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vz888" podStartSLOduration=65.663456682 podStartE2EDuration="1m14.545090819s" totalImagesPullingTime="8.881634137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:36.54446627 +0000 UTC m=+75.045546997" watchObservedRunningTime="2026-07-16 20:34:36.545090819 +0000 UTC m=+75.046171555" Jul 16 20:34:37.872967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:37.872932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:37.929936 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:37.929901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:38.299551 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:38.299515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:39.681815 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:39.681779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:39.782433 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:39.782397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:40.522413 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:40.522377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:41.357099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:41.357064 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:41.415566 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:41.415539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:46.414987 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:46.414873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:47.438591 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:47.438565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657968875d-v8h6h" Jul 16 20:34:47.566668 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:47.566627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:47.671077 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:47.671044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.436029 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.435984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.533431 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.533397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.639966 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.639930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.658224 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.658196 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5"] Jul 16 20:34:48.661590 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.661570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.666605 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.666585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 20:34:48.667152 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.667129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 20:34:48.667627 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.667608 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 20:34:48.667709 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.667629 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 20:34:48.668141 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.668125 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 20:34:48.668238 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.668222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kvmd8\"" Jul 16 20:34:48.676621 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.676594 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5"] Jul 16 20:34:48.683796 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.683772 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gckch"] Jul 16 20:34:48.685921 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.685905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.688867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.688809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-trqdw\"" Jul 16 20:34:48.688867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.688833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 20:34:48.688867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.688850 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 20:34:48.689055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.688904 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 20:34:48.729457 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729457 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-wtmp\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-root\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-tls\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-sys\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729690 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-textfile\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.729962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100cea57-ab54-439a-a514-492e7fff2f42-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.729962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkh6\" (UniqueName: \"kubernetes.io/projected/100cea57-ab54-439a-a514-492e7fff2f42-kube-api-access-zlkh6\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.729962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-metrics-client-ca\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.729962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.729910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbcx\" (UniqueName: \"kubernetes.io/projected/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-kube-api-access-mtbcx\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.807538 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.807499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.830577 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.830744 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100cea57-ab54-439a-a514-492e7fff2f42-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.830744 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkh6\" (UniqueName: \"kubernetes.io/projected/100cea57-ab54-439a-a514-492e7fff2f42-kube-api-access-zlkh6\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.830744 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:48.830712 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 20:34:48.830744 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-metrics-client-ca\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.830937 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:48.830781 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls podName:100cea57-ab54-439a-a514-492e7fff2f42 nodeName:}" failed. No retries permitted until 2026-07-16 20:34:49.330759128 +0000 UTC m=+87.831839835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-tc7r5" (UID: "100cea57-ab54-439a-a514-492e7fff2f42") : secret "openshift-state-metrics-tls" not found Jul 16 20:34:48.830937 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbcx\" (UniqueName: \"kubernetes.io/projected/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-kube-api-access-mtbcx\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.830937 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.830981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-wtmp\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-root\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831245 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.831245 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-tls\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831245 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-sys\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831245 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-wtmp\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831245 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-textfile\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831377 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100cea57-ab54-439a-a514-492e7fff2f42-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-metrics-client-ca\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-sys\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-root\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.831515 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.831512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-textfile\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.833525 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.833497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.833631 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.833585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-node-exporter-tls\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.833676 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.833649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.843859 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.843829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkh6\" (UniqueName: \"kubernetes.io/projected/100cea57-ab54-439a-a514-492e7fff2f42-kube-api-access-zlkh6\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:48.843941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.843878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbcx\" (UniqueName: \"kubernetes.io/projected/d308c4d9-cd87-4ee2-9e04-890e7afb2f3b-kube-api-access-mtbcx\") pod \"node-exporter-gckch\" (UID: \"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b\") " pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:48.878059 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.878015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.935877 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.935849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:48.996352 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:48.996326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gckch" Jul 16 20:34:49.006336 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:49.006288 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd308c4d9_cd87_4ee2_9e04_890e7afb2f3b.slice/crio-a23dc2e06d2d95f97265e1d5d5e42a4249d2c9d3e6393458bc858e569f443154 WatchSource:0}: Error finding container a23dc2e06d2d95f97265e1d5d5e42a4249d2c9d3e6393458bc858e569f443154: Status 404 returned error can't find the container with id a23dc2e06d2d95f97265e1d5d5e42a4249d2c9d3e6393458bc858e569f443154 Jul 16 20:34:49.154378 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.154351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:49.335546 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.335446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:49.338039 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.338012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100cea57-ab54-439a-a514-492e7fff2f42-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tc7r5\" (UID: \"100cea57-ab54-439a-a514-492e7fff2f42\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:49.516115 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.516074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gckch" event={"ID":"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b","Type":"ContainerStarted","Data":"a23dc2e06d2d95f97265e1d5d5e42a4249d2c9d3e6393458bc858e569f443154"} Jul 16 20:34:49.572466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.572418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" Jul 16 20:34:49.685755 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.685706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:49.714616 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.714583 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5"] Jul 16 20:34:49.717512 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:49.717482 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100cea57_ab54_439a_a514_492e7fff2f42.slice/crio-7452417cdf504add8e818f52ee6cea30a23a604d410bf9001fd1305ab10a2778 WatchSource:0}: Error finding container 7452417cdf504add8e818f52ee6cea30a23a604d410bf9001fd1305ab10a2778: Status 404 returned error can't find the container with id 7452417cdf504add8e818f52ee6cea30a23a604d410bf9001fd1305ab10a2778 Jul 16 20:34:49.772169 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.772139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:49.775648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.775628 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 20:34:49.778209 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.778189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.780859 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.780839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 20:34:49.780955 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.780882 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 20:34:49.782249 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zpk48\"" Jul 16 20:34:49.782470 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782454 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 20:34:49.782621 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 20:34:49.782674 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 20:34:49.782758 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 20:34:49.782841 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.782827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 20:34:49.783108 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.783091 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 20:34:49.783164 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.783091 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 20:34:49.796667 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.796578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.838977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-out\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-web-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.839415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.839285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6472\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-kube-api-access-r6472\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.920421 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.920288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:49.940588 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.940756 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-web-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.940756 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.940756 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.940756 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.940756 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6472\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-kube-api-access-r6472\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.940932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-out\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.941577 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:34:49.941085 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle podName:7aa42b13-9d36-432b-8cc7-e8f7219114bb nodeName:}" failed. No retries permitted until 2026-07-16 20:34:50.441059939 +0000 UTC m=+88.942140659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7aa42b13-9d36-432b-8cc7-e8f7219114bb") : configmap references non-existent config key: ca-bundle.crt Jul 16 20:34:49.941577 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.941494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.942224 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.942201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.944095 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.944066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.944807 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.944765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-out\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.945063 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.945007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.945361 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.945332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.945568 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.945461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.945938 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.945908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.946000 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.945975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-web-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.946287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.946246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.947890 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.947869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7aa42b13-9d36-432b-8cc7-e8f7219114bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:49.950872 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:49.950831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6472\" (UniqueName: \"kubernetes.io/projected/7aa42b13-9d36-432b-8cc7-e8f7219114bb-kube-api-access-r6472\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:50.236428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.236405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:50.445128 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.445097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:50.445804 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.445786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa42b13-9d36-432b-8cc7-e8f7219114bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7aa42b13-9d36-432b-8cc7-e8f7219114bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:50.521452 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.521374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" event={"ID":"100cea57-ab54-439a-a514-492e7fff2f42","Type":"ContainerStarted","Data":"b118d33ca523060f72c8c8503c2b68ccb514f2a163387ee5068b2ae6521ccb01"} Jul 16 20:34:50.521452 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.521443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" event={"ID":"100cea57-ab54-439a-a514-492e7fff2f42","Type":"ContainerStarted","Data":"1ac59614b6b8016ddf4035b6cc4f9ebd3adc6fb5c5c5f89ac2cd570fe5f80e39"} Jul 16 20:34:50.521633 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.521468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" event={"ID":"100cea57-ab54-439a-a514-492e7fff2f42","Type":"ContainerStarted","Data":"7452417cdf504add8e818f52ee6cea30a23a604d410bf9001fd1305ab10a2778"} Jul 16 20:34:50.522982 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.522957 2564 generic.go:356] "Generic (PLEG): container finished" podID="d308c4d9-cd87-4ee2-9e04-890e7afb2f3b" containerID="80e1355dc5ac0166075f59be1c6d02e4c08c7c555623c4e93e76fd763f0daf66" exitCode=0 Jul 16 20:34:50.523103 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.523016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gckch" event={"ID":"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b","Type":"ContainerDied","Data":"80e1355dc5ac0166075f59be1c6d02e4c08c7c555623c4e93e76fd763f0daf66"} Jul 16 20:34:50.691000 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.690967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 20:34:50.836758 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:50.836709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 20:34:50.840723 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:34:50.840700 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa42b13_9d36_432b_8cc7_e8f7219114bb.slice/crio-279e7e1f2e0127f0f2790e6fe18d8a5cf5460f02af1bb144bf4a29979f9944dd WatchSource:0}: Error finding container 279e7e1f2e0127f0f2790e6fe18d8a5cf5460f02af1bb144bf4a29979f9944dd: Status 404 returned error can't find the container with id 279e7e1f2e0127f0f2790e6fe18d8a5cf5460f02af1bb144bf4a29979f9944dd Jul 16 20:34:51.132987 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.132911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:51.233937 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.233905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:51.529413 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.529375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gckch" event={"ID":"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b","Type":"ContainerStarted","Data":"86d8829b913f730821ecdc145f9b4b7556d3d1c434a13c8c349488a296d45a58"} Jul 16 20:34:51.529583 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.529436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gckch" event={"ID":"d308c4d9-cd87-4ee2-9e04-890e7afb2f3b","Type":"ContainerStarted","Data":"6f71b38dbb06b6d93ad624a26a4752cd27d9bbb7fbaf6cffbf102e6d4d2e8a0e"} Jul 16 20:34:51.530962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.530940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"279e7e1f2e0127f0f2790e6fe18d8a5cf5460f02af1bb144bf4a29979f9944dd"} Jul 16 20:34:51.549493 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.549435 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gckch" podStartSLOduration=2.255441412 podStartE2EDuration="3.549413239s" totalImagesPullingTime="1.22398541s" totalInitContainerRuntime="69.986417ms" isStatefulPod=true podCreationTimestamp="2026-07-16 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:51.547911374 +0000 UTC m=+90.048992100" watchObservedRunningTime="2026-07-16 20:34:51.549413239 +0000 UTC m=+90.050493962" Jul 16 20:34:51.599194 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.599160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:51.689824 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.689788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:51.805651 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.805569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:51.915552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:51.915481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:53.489598 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.489561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:53.541250 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.541210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" event={"ID":"100cea57-ab54-439a-a514-492e7fff2f42","Type":"ContainerStarted","Data":"471eee535f907a1eb3eb9ff927b8173aa794af5c619e2bd9d628192cca7cb5e2"} Jul 16 20:34:53.542835 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.542807 2564 generic.go:356] "Generic (PLEG): container finished" podID="7aa42b13-9d36-432b-8cc7-e8f7219114bb" containerID="50c1ba95f893294a80e96ee2bdde882c84973654b4f1499c99ecb82fdcf5cbfe" exitCode=0 Jul 16 20:34:53.542957 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.542858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerDied","Data":"50c1ba95f893294a80e96ee2bdde882c84973654b4f1499c99ecb82fdcf5cbfe"} Jul 16 20:34:53.550577 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.550553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:53.559731 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.559680 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tc7r5" podStartSLOduration=2.56252074 podStartE2EDuration="5.559659349s" totalImagesPullingTime="2.997138609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:53.559093402 +0000 UTC m=+92.060174139" watchObservedRunningTime="2026-07-16 20:34:53.559659349 +0000 UTC m=+92.060740076" Jul 16 20:34:53.631098 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.631057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:53.733877 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.733847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:53.932792 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:53.932717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:54.972208 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:54.972178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:56.557490 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:56.557456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"8441a4bdb492e9282973166f5cc04ad23b04f5878a2f7570c0a223e340ed55bc"} Jul 16 20:34:56.557855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:56.557521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"d886ca38d701b3c0ec7f2f7045c0bc0df487c2fd94c59f676e7541fee777a018"} Jul 16 20:34:56.557855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:56.557549 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"b258d5999abe7ffa37be68c38a0e063a6369e1d5955426909477559cb2014961"} Jul 16 20:34:56.557855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:56.557572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"3b9e748af4c6dd7818fa4186bee716b7a4d599d3523404a4728e6f42de5f2cb5"} Jul 16 20:34:56.557855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:56.557595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"2f3d5aa7fb2aa9278054743e20e310d47cf98171722078318db79d5785c1d874"} Jul 16 20:34:57.559465 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:57.559432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:34:57.563982 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:57.563957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7aa42b13-9d36-432b-8cc7-e8f7219114bb","Type":"ContainerStarted","Data":"9833f58af420083a050c3b3d80f3cbd8321c240d1b8f2e6a90f795ffe6dc6dc8"} Jul 16 20:34:57.591024 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:57.590968 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.549247664 podStartE2EDuration="8.590953489s" totalImagesPullingTime="5.023208197s" totalInitContainerRuntime="18.497628ms" isStatefulPod=false podCreationTimestamp="2026-07-16 20:34:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:57.589837028 +0000 UTC m=+96.090917754" watchObservedRunningTime="2026-07-16 20:34:57.590953489 +0000 UTC m=+96.092034214" Jul 16 20:34:57.660694 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:34:57.660660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:00.224874 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:00.224838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:00.331080 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:00.331031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:03.431650 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:03.431609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:03.529234 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:03.529205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:04.444973 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:04.444938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:04.548371 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:04.548333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:04.601538 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:04.601506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:07.480355 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:07.480327 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jldkm" Jul 16 20:35:18.433800 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:18.433761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:18.534125 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:18.534087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:29.120428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:29.120389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:29.218810 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:29.218769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:29.273413 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:29.273382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:35:49.988546 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:35:49.988511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:07.930708 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:07.930666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:08.218101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:08.218024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:13.069986 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.069903 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx"] Jul 16 20:36:13.073041 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.073024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.079531 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079508 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 20:36:13.079720 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 20:36:13.079720 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079711 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 20:36:13.079836 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079711 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 20:36:13.079836 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079778 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lm4qm\"" Jul 16 20:36:13.079836 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.079830 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 20:36:13.099864 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.099821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 20:36:13.107207 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.107176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx"] Jul 16 20:36:13.182648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-serving-certs-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-metrics-client-ca\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pbz\" (UniqueName: \"kubernetes.io/projected/ca53ca11-351a-4add-b2b7-4d17c092d0fa-kube-api-access-95pbz\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.182904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-federate-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.183100 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.182918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.283922 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.283880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284004 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.283932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284004 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.283957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-federate-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284004 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.283977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.284014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-serving-certs-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.284034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.284068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-metrics-client-ca\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.284111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pbz\" (UniqueName: \"kubernetes.io/projected/ca53ca11-351a-4add-b2b7-4d17c092d0fa-kube-api-access-95pbz\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.284900 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.284835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-serving-certs-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.285175 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.285151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.285279 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.285254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca53ca11-351a-4add-b2b7-4d17c092d0fa-metrics-client-ca\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.286749 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.286725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.286826 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.286801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-federate-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.286903 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.286886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-secret-telemeter-client\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.286994 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.286973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca53ca11-351a-4add-b2b7-4d17c092d0fa-telemeter-client-tls\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.293332 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.293286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pbz\" (UniqueName: \"kubernetes.io/projected/ca53ca11-351a-4add-b2b7-4d17c092d0fa-kube-api-access-95pbz\") pod \"telemeter-client-84d55bcd7c-4gwtx\" (UID: \"ca53ca11-351a-4add-b2b7-4d17c092d0fa\") " pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.381580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.381488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" Jul 16 20:36:13.519480 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.519446 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx"] Jul 16 20:36:13.523054 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:36:13.523027 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca53ca11_351a_4add_b2b7_4d17c092d0fa.slice/crio-8e857b2bafe2666f3b0a52362557aa770a3f22a5b78efc42fe7cd2108aab220a WatchSource:0}: Error finding container 8e857b2bafe2666f3b0a52362557aa770a3f22a5b78efc42fe7cd2108aab220a: Status 404 returned error can't find the container with id 8e857b2bafe2666f3b0a52362557aa770a3f22a5b78efc42fe7cd2108aab220a Jul 16 20:36:13.824952 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:13.824915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" event={"ID":"ca53ca11-351a-4add-b2b7-4d17c092d0fa","Type":"ContainerStarted","Data":"8e857b2bafe2666f3b0a52362557aa770a3f22a5b78efc42fe7cd2108aab220a"} Jul 16 20:36:15.834119 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:15.834080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" event={"ID":"ca53ca11-351a-4add-b2b7-4d17c092d0fa","Type":"ContainerStarted","Data":"7ac73bca200eb594968c3203af5896fd86f9fa5a1a75474a66203dafd287e785"} Jul 16 20:36:15.834525 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:15.834136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" event={"ID":"ca53ca11-351a-4add-b2b7-4d17c092d0fa","Type":"ContainerStarted","Data":"f108618f342ec9f8a14f7ebd246f54b3905f00e304a812fb65b152ddbd94abca"} Jul 16 20:36:15.834525 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:15.834165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" event={"ID":"ca53ca11-351a-4add-b2b7-4d17c092d0fa","Type":"ContainerStarted","Data":"60cdee2c6764da07b0e31e2276e97575d8786068f2ec4c0f75118f3c51d4d033"} Jul 16 20:36:15.859367 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:15.859318 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84d55bcd7c-4gwtx" podStartSLOduration=1.217819701 podStartE2EDuration="2.859281716s" totalImagesPullingTime="1.641462015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:36:15.85844824 +0000 UTC m=+174.359528968" watchObservedRunningTime="2026-07-16 20:36:15.859281716 +0000 UTC m=+174.360362442" Jul 16 20:36:16.611713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:16.611685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:16.663548 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:16.663519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:16.750838 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:16.750802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:16.805997 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:16.805972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:17.352739 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:17.352709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:17.453037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:17.453003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:24.734621 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:24.734588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:24.785427 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:24.785399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:35.169637 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:35.169601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:35.222533 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:35.222499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:36.819859 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:36.819828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:38.027294 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:38.027255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:39.166277 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:39.166236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:48.433959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:48.433921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:48.534567 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:48.534539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:56.827222 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:56.827187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:36:56.972666 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:36:56.972637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:33.931741 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:33.931703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:34.025654 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:34.025623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:55.545440 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:55.545367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:56.347313 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:56.347270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:59.301752 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:59.301717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:37:59.414714 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:37:59.414677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:00.418867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:00.418835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:00.528789 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:00.528757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:01.141684 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:01.141648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:01.249909 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:01.249880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:03.118025 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:03.117991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:03.509926 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:03.509887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:05.449122 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:05.449085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:05.511750 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:05.511713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:06.135555 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:06.135525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:06.196285 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:06.196254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:11.111387 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:11.111353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:11.211135 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:11.211105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:11.720810 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:11.720780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:12.500935 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.500901 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sxd9x"] Jul 16 20:38:12.502728 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.502713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.505354 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.505330 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 20:38:12.506502 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.506479 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m8vs6\"" Jul 16 20:38:12.506602 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.506479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 20:38:12.513512 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.513490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sxd9x"] Jul 16 20:38:12.532534 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.532503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-bound-sa-token\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.532717 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.532565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2g6t\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-kube-api-access-t2g6t\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.633713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.633662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2g6t\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-kube-api-access-t2g6t\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.633889 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.633735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-bound-sa-token\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.641919 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.641882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-bound-sa-token\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.642039 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.641973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2g6t\" (UniqueName: \"kubernetes.io/projected/bf46c782-d728-422f-8676-ab05677076b0-kube-api-access-t2g6t\") pod \"cert-manager-55b4d954d5-sxd9x\" (UID: \"bf46c782-d728-422f-8676-ab05677076b0\") " pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.811492 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.811399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sxd9x" Jul 16 20:38:12.944025 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:12.943994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sxd9x"] Jul 16 20:38:12.946890 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:12.946865 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf46c782_d728_422f_8676_ab05677076b0.slice/crio-d3e6046b5052ccbe6821fad4839fa77cd870a0bdbbaa1c02a85e9893fd66c5f2 WatchSource:0}: Error finding container d3e6046b5052ccbe6821fad4839fa77cd870a0bdbbaa1c02a85e9893fd66c5f2: Status 404 returned error can't find the container with id d3e6046b5052ccbe6821fad4839fa77cd870a0bdbbaa1c02a85e9893fd66c5f2 Jul 16 20:38:13.247227 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.247189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sxd9x" event={"ID":"bf46c782-d728-422f-8676-ab05677076b0","Type":"ContainerStarted","Data":"d3e6046b5052ccbe6821fad4839fa77cd870a0bdbbaa1c02a85e9893fd66c5f2"} Jul 16 20:38:13.946417 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.946378 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf"] Jul 16 20:38:13.948645 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.948615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:13.951349 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.951323 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 20:38:13.952468 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.952446 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qhhwm\"" Jul 16 20:38:13.952583 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.952490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 20:38:13.959379 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:13.959356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf"] Jul 16 20:38:14.044693 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.044664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed5ffd9-b170-495f-b06f-67464b9c92af-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.044902 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.044726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nchb\" (UniqueName: \"kubernetes.io/projected/3ed5ffd9-b170-495f-b06f-67464b9c92af-kube-api-access-2nchb\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.145247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.145208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed5ffd9-b170-495f-b06f-67464b9c92af-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.145445 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.145295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nchb\" (UniqueName: \"kubernetes.io/projected/3ed5ffd9-b170-495f-b06f-67464b9c92af-kube-api-access-2nchb\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.145685 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.145662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed5ffd9-b170-495f-b06f-67464b9c92af-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.154347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.154295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nchb\" (UniqueName: \"kubernetes.io/projected/3ed5ffd9-b170-495f-b06f-67464b9c92af-kube-api-access-2nchb\") pod \"openshift-lws-operator-fd8ccff4c-gkxzf\" (UID: \"3ed5ffd9-b170-495f-b06f-67464b9c92af\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:14.261447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:14.261411 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" Jul 16 20:38:15.826464 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:15.826275 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf"] Jul 16 20:38:15.829343 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:15.829313 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ed5ffd9_b170_495f_b06f_67464b9c92af.slice/crio-8bb44470cabcd9abc21afbee5c644c45ddecb4d28320c7ec8ccb023cfa3fc893 WatchSource:0}: Error finding container 8bb44470cabcd9abc21afbee5c644c45ddecb4d28320c7ec8ccb023cfa3fc893: Status 404 returned error can't find the container with id 8bb44470cabcd9abc21afbee5c644c45ddecb4d28320c7ec8ccb023cfa3fc893 Jul 16 20:38:16.263474 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:16.263435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" event={"ID":"3ed5ffd9-b170-495f-b06f-67464b9c92af","Type":"ContainerStarted","Data":"8bb44470cabcd9abc21afbee5c644c45ddecb4d28320c7ec8ccb023cfa3fc893"} Jul 16 20:38:16.265467 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:16.265437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sxd9x" event={"ID":"bf46c782-d728-422f-8676-ab05677076b0","Type":"ContainerStarted","Data":"c00ebe23339f8be35945aa073d52f16fc90773d9fca03fbc8a03ae64e2d48609"} Jul 16 20:38:16.282166 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:16.282108 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sxd9x" podStartSLOduration=1.508320432 podStartE2EDuration="4.282088587s" totalImagesPullingTime="2.773768155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:38:16.281152782 +0000 UTC m=+294.782233508" watchObservedRunningTime="2026-07-16 20:38:16.282088587 +0000 UTC m=+294.783169313" Jul 16 20:38:18.275248 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:18.275214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" event={"ID":"3ed5ffd9-b170-495f-b06f-67464b9c92af","Type":"ContainerStarted","Data":"80f4a15f097fadedc0422935d15dad8cf1c3ef3a53ba4e8bf3098c34d7a34d66"} Jul 16 20:38:18.291001 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:18.290904 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkxzf" podStartSLOduration=3.085452657 podStartE2EDuration="5.290883469s" totalImagesPullingTime="2.205430812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:38:18.290617717 +0000 UTC m=+296.791698445" watchObservedRunningTime="2026-07-16 20:38:18.290883469 +0000 UTC m=+296.791964197" Jul 16 20:38:18.971154 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:18.971119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:20.722758 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:20.722700 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:21.375170 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:21.375129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:21.872444 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:21.872413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:38:21.876830 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:21.876807 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:38:21.880261 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:21.880241 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 20:38:29.962603 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:29.962552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:30.064826 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:30.064783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:33.438837 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:33.438809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:33.552904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:33.552868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:33.937892 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:33.937859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.183852 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.183815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.295522 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.295485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.340096 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.340061 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf"] Jul 16 20:38:34.399428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.399389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf"] Jul 16 20:38:34.399580 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.399521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.402779 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.402750 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 20:38:34.402967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.402758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 20:38:34.402967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.402869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 20:38:34.402967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.402929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vwh9v\"" Jul 16 20:38:34.403170 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.403154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 20:38:34.520479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.520445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.520479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.520481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.520891 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.520563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.520891 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.520605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.520891 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.520624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5fr\" (UniqueName: \"kubernetes.io/projected/af6d3e47-493e-4830-8a86-8d173191a8e4-kube-api-access-ph5fr\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.526533 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.526505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.621976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.621891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.621976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.621928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.621976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.621971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.622237 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.621993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.622237 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.622009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5fr\" (UniqueName: \"kubernetes.io/projected/af6d3e47-493e-4830-8a86-8d173191a8e4-kube-api-access-ph5fr\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.622390 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.622281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-manifests\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.624539 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.624510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6d3e47-493e-4830-8a86-8d173191a8e4-tmp\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.624705 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.624680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.624843 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.624820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af6d3e47-493e-4830-8a86-8d173191a8e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.627376 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.627351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.630680 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.630658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5fr\" (UniqueName: \"kubernetes.io/projected/af6d3e47-493e-4830-8a86-8d173191a8e4-kube-api-access-ph5fr\") pod \"opendatahub-operator-controller-manager-6d6477549b-bw6kf\" (UID: \"af6d3e47-493e-4830-8a86-8d173191a8e4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.710208 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.710174 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:34.731250 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.731217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.852253 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.852219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:34.870415 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.870334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf"] Jul 16 20:38:34.872546 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:34.872492 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6d3e47_493e_4830_8a86_8d173191a8e4.slice/crio-2ec04a39f0f1817ce51eabc8140761d4a78b366d92f2ff31c5410fc2cea2b175 WatchSource:0}: Error finding container 2ec04a39f0f1817ce51eabc8140761d4a78b366d92f2ff31c5410fc2cea2b175: Status 404 returned error can't find the container with id 2ec04a39f0f1817ce51eabc8140761d4a78b366d92f2ff31c5410fc2cea2b175 Jul 16 20:38:34.874502 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.874483 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 20:38:34.954505 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:34.954478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:35.052777 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:35.052745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:35.337809 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:35.337775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" event={"ID":"af6d3e47-493e-4830-8a86-8d173191a8e4","Type":"ContainerStarted","Data":"2ec04a39f0f1817ce51eabc8140761d4a78b366d92f2ff31c5410fc2cea2b175"} Jul 16 20:38:38.352748 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:38.352715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" event={"ID":"af6d3e47-493e-4830-8a86-8d173191a8e4","Type":"ContainerStarted","Data":"f6864c99c5f3d7062e795b03f9080b4c2bdabca2435bad5647d2874eb91842f7"} Jul 16 20:38:39.358296 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:39.358265 2564 generic.go:356] "Generic (PLEG): container finished" podID="af6d3e47-493e-4830-8a86-8d173191a8e4" containerID="f6864c99c5f3d7062e795b03f9080b4c2bdabca2435bad5647d2874eb91842f7" exitCode=0 Jul 16 20:38:39.358801 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:39.358351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" event={"ID":"af6d3e47-493e-4830-8a86-8d173191a8e4","Type":"ContainerDied","Data":"f6864c99c5f3d7062e795b03f9080b4c2bdabca2435bad5647d2874eb91842f7"} Jul 16 20:38:39.687939 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:39.687901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:39.793611 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:39.793573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:40.363567 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:40.363529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" event={"ID":"af6d3e47-493e-4830-8a86-8d173191a8e4","Type":"ContainerStarted","Data":"5860b04c824f818b466e633e94971a9876cb04c74e67da95306d760b0674f8cd"} Jul 16 20:38:40.363960 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:40.363689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:40.386814 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:40.386756 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" podStartSLOduration=2.1377949 podStartE2EDuration="6.386740984s" totalImagesPullingTime="3.155576478s" totalInitContainerRuntime="1.093369606s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:38:40.384941838 +0000 UTC m=+318.886022578" watchObservedRunningTime="2026-07-16 20:38:40.386740984 +0000 UTC m=+318.887821714" Jul 16 20:38:51.369949 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:51.369914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6477549b-bw6kf" Jul 16 20:38:54.473927 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:54.473891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:54.613497 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:54.613468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:55.197120 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.197075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:55.258358 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.258322 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn"] Jul 16 20:38:55.265395 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.265372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.270551 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.270530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mb988\"" Jul 16 20:38:55.270816 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.270803 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 20:38:55.271625 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.271605 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 20:38:55.272377 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.272358 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 20:38:55.272764 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.272746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 20:38:55.275587 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.275565 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn"] Jul 16 20:38:55.294501 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.294474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8675372-fc09-480b-8fad-dead11a91eca-tmp\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.294682 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.294538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbhr\" (UniqueName: \"kubernetes.io/projected/a8675372-fc09-480b-8fad-dead11a91eca-kube-api-access-fmbhr\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.294682 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.294586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8675372-fc09-480b-8fad-dead11a91eca-tls-certs\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.309169 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.309132 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x5l4z"] Jul 16 20:38:55.312424 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.312400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.315145 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.315119 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 20:38:55.315283 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.315182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t57ch\"" Jul 16 20:38:55.325609 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.325579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x5l4z"] Jul 16 20:38:55.344177 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.344154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:55.395741 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.395709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8675372-fc09-480b-8fad-dead11a91eca-tls-certs\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.395922 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.395776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8675372-fc09-480b-8fad-dead11a91eca-tmp\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.395922 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.395815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.395922 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.395865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbhr\" (UniqueName: \"kubernetes.io/projected/a8675372-fc09-480b-8fad-dead11a91eca-kube-api-access-fmbhr\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.396050 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.395924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/ca05d45f-a8fb-452d-b979-f2905a1eb08e-kube-api-access-2zxjf\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.398336 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.398283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8675372-fc09-480b-8fad-dead11a91eca-tmp\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.398927 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.398875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8675372-fc09-480b-8fad-dead11a91eca-tls-certs\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.407031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.406995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbhr\" (UniqueName: \"kubernetes.io/projected/a8675372-fc09-480b-8fad-dead11a91eca-kube-api-access-fmbhr\") pod \"kube-auth-proxy-559f5f697c-vwjwn\" (UID: \"a8675372-fc09-480b-8fad-dead11a91eca\") " pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.496500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.496457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/ca05d45f-a8fb-452d-b979-f2905a1eb08e-kube-api-access-2zxjf\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.496966 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.496568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.496966 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:38:55.496679 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 16 20:38:55.496966 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:38:55.496738 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert podName:ca05d45f-a8fb-452d-b979-f2905a1eb08e nodeName:}" failed. No retries permitted until 2026-07-16 20:38:55.996716611 +0000 UTC m=+334.497797316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert") pod "odh-model-controller-7998c67dc4-x5l4z" (UID: "ca05d45f-a8fb-452d-b979-f2905a1eb08e") : secret "odh-model-controller-webhook-cert" not found Jul 16 20:38:55.507858 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.507828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/ca05d45f-a8fb-452d-b979-f2905a1eb08e-kube-api-access-2zxjf\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:55.579280 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.579238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" Jul 16 20:38:55.734567 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.734537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn"] Jul 16 20:38:55.736499 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:55.736467 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8675372_fc09_480b_8fad_dead11a91eca.slice/crio-6af6f768132f4fa96f6fd2d1da013fcdfbea0fa74aed2347ec24b412297ce740 WatchSource:0}: Error finding container 6af6f768132f4fa96f6fd2d1da013fcdfbea0fa74aed2347ec24b412297ce740: Status 404 returned error can't find the container with id 6af6f768132f4fa96f6fd2d1da013fcdfbea0fa74aed2347ec24b412297ce740 Jul 16 20:38:55.800037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.799951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:55.879374 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:55.879340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:56.000867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.000835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:56.004025 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.003989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca05d45f-a8fb-452d-b979-f2905a1eb08e-cert\") pod \"odh-model-controller-7998c67dc4-x5l4z\" (UID: \"ca05d45f-a8fb-452d-b979-f2905a1eb08e\") " pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:56.184138 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.184061 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j2k9j"] Jul 16 20:38:56.188629 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.188612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.191479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.191456 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 16 20:38:56.202404 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.202373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twf77\" (UniqueName: \"kubernetes.io/projected/c8eb74e0-600a-4af0-96b3-47143c987d14-kube-api-access-twf77\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.202552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.202425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.205769 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.205744 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j2k9j"] Jul 16 20:38:56.217453 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.217426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:56.226353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.226326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:38:56.303323 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.302977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twf77\" (UniqueName: \"kubernetes.io/projected/c8eb74e0-600a-4af0-96b3-47143c987d14-kube-api-access-twf77\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.303323 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.303036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.303323 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:38:56.303202 2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 16 20:38:56.303323 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:38:56.303266 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs podName:c8eb74e0-600a-4af0-96b3-47143c987d14 nodeName:}" failed. No retries permitted until 2026-07-16 20:38:56.803242629 +0000 UTC m=+335.304323354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs") pod "model-serving-api-54d89bd5fc-j2k9j" (UID: "c8eb74e0-600a-4af0-96b3-47143c987d14") : secret "model-serving-api-tls" not found Jul 16 20:38:56.315883 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.315848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twf77\" (UniqueName: \"kubernetes.io/projected/c8eb74e0-600a-4af0-96b3-47143c987d14-kube-api-access-twf77\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.404747 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.404715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-x5l4z"] Jul 16 20:38:56.415169 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:56.415125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca05d45f_a8fb_452d_b979_f2905a1eb08e.slice/crio-eeb126c56445ea8e6fb9dd4c37e7032ce8f9988e13843243814cfee34807302d WatchSource:0}: Error finding container eeb126c56445ea8e6fb9dd4c37e7032ce8f9988e13843243814cfee34807302d: Status 404 returned error can't find the container with id eeb126c56445ea8e6fb9dd4c37e7032ce8f9988e13843243814cfee34807302d Jul 16 20:38:56.440235 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.440153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" event={"ID":"a8675372-fc09-480b-8fad-dead11a91eca","Type":"ContainerStarted","Data":"6af6f768132f4fa96f6fd2d1da013fcdfbea0fa74aed2347ec24b412297ce740"} Jul 16 20:38:56.441327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.441282 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" event={"ID":"ca05d45f-a8fb-452d-b979-f2905a1eb08e","Type":"ContainerStarted","Data":"eeb126c56445ea8e6fb9dd4c37e7032ce8f9988e13843243814cfee34807302d"} Jul 16 20:38:56.808222 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.808181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.811404 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.811352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8eb74e0-600a-4af0-96b3-47143c987d14-tls-certs\") pod \"model-serving-api-54d89bd5fc-j2k9j\" (UID: \"c8eb74e0-600a-4af0-96b3-47143c987d14\") " pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:56.812852 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:56.812823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:57.099659 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.099578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:38:57.190913 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.190877 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:38:57.195468 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.194658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.197523 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.197493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:57.198329 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.197761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 20:38:57.198329 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.198287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 20:38:57.199409 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.198527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-42cc9\"" Jul 16 20:38:57.199409 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.198751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 20:38:57.199409 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.198975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 20:38:57.211252 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211407 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211407 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211529 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7c5z\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211529 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211529 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.211529 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.211523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.212836 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.212030 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:38:57.288077 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.287999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j2k9j"] Jul 16 20:38:57.290686 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.290498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:57.292459 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:57.292162 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eb74e0_600a_4af0_96b3_47143c987d14.slice/crio-198967c9647cf5d8ae50a35a74bfa21ed0318870c72aa30d2054d0fae0a5a44c WatchSource:0}: Error finding container 198967c9647cf5d8ae50a35a74bfa21ed0318870c72aa30d2054d0fae0a5a44c: Status 404 returned error can't find the container with id 198967c9647cf5d8ae50a35a74bfa21ed0318870c72aa30d2054d0fae0a5a44c Jul 16 20:38:57.312725 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.312893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c5z\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.312893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.312893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.312893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.312893 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.313149 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.312908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.314419 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.314393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.319130 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.319079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.319240 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.319159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.320173 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.320144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.320821 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.320799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.324994 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.324868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.327142 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.327115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c5z\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z\") pod \"istiod-openshift-gateway-7574459656-mn26j\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.449923 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.449835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" event={"ID":"c8eb74e0-600a-4af0-96b3-47143c987d14","Type":"ContainerStarted","Data":"198967c9647cf5d8ae50a35a74bfa21ed0318870c72aa30d2054d0fae0a5a44c"} Jul 16 20:38:57.509626 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.509333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:38:57.902251 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:57.902209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:38:58.733323 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:58.733274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:38:59.337399 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:38:59.337361 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1565d4e9_2767_43cb_96bc_766cb24fe6fe.slice/crio-f3a2b884573562866b8ee01b0bc2e7fc0d0ac309e722e38a66607a0a50943b80 WatchSource:0}: Error finding container f3a2b884573562866b8ee01b0bc2e7fc0d0ac309e722e38a66607a0a50943b80: Status 404 returned error can't find the container with id f3a2b884573562866b8ee01b0bc2e7fc0d0ac309e722e38a66607a0a50943b80 Jul 16 20:38:59.461902 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:38:59.461867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" event={"ID":"1565d4e9-2767-43cb-96bc-766cb24fe6fe","Type":"ContainerStarted","Data":"f3a2b884573562866b8ee01b0bc2e7fc0d0ac309e722e38a66607a0a50943b80"} Jul 16 20:39:01.478520 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.478480 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" event={"ID":"c8eb74e0-600a-4af0-96b3-47143c987d14","Type":"ContainerStarted","Data":"550d7ef08d6add6dc8cb796179469edaba63d5321394959cabf18c73112b5a92"} Jul 16 20:39:01.478972 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.478677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:39:01.480150 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.480125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" event={"ID":"a8675372-fc09-480b-8fad-dead11a91eca","Type":"ContainerStarted","Data":"af01a847938698ff362cbf336a2ba368c12d4dbbc202307f99df23b3c829a8e2"} Jul 16 20:39:01.481763 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.481736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" event={"ID":"ca05d45f-a8fb-452d-b979-f2905a1eb08e","Type":"ContainerStarted","Data":"4b42d62e2e63219a7a50cce71fe84fa71425dad3c01bac1f4c6216fccdeee771"} Jul 16 20:39:01.481874 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.481799 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:39:01.531726 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.531671 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-559f5f697c-vwjwn" podStartSLOduration=1.8308168390000001 podStartE2EDuration="6.531653249s" totalImagesPullingTime="4.70083641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:01.531477578 +0000 UTC m=+340.032558296" watchObservedRunningTime="2026-07-16 20:39:01.531653249 +0000 UTC m=+340.032733975" Jul 16 20:39:01.531932 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.531772 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" podStartSLOduration=2.390268752 podStartE2EDuration="5.531763078s" totalImagesPullingTime="3.141494326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:01.504292262 +0000 UTC m=+340.005372989" watchObservedRunningTime="2026-07-16 20:39:01.531763078 +0000 UTC m=+340.032843805" Jul 16 20:39:01.562126 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:01.562055 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" podStartSLOduration=2.546601997 podStartE2EDuration="6.562023455s" totalImagesPullingTime="4.015421458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:01.561200458 +0000 UTC m=+340.062281183" watchObservedRunningTime="2026-07-16 20:39:01.562023455 +0000 UTC m=+340.063104265" Jul 16 20:39:04.364417 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:04.364369 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:39:04.364725 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:04.364461 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:39:04.498904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:04.498861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" event={"ID":"1565d4e9-2767-43cb-96bc-766cb24fe6fe","Type":"ContainerStarted","Data":"2fca0b2f06dc8d29eab0482cf556fbf3e4b1e39efcd82cd07a439579fd1d51b0"} Jul 16 20:39:04.499070 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:04.498935 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:39:04.528909 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:04.528852 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" podStartSLOduration=2.50479977 podStartE2EDuration="7.528836674s" totalImagesPullingTime="5.024036904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:04.526996416 +0000 UTC m=+343.028077141" watchObservedRunningTime="2026-07-16 20:39:04.528836674 +0000 UTC m=+343.029917400" Jul 16 20:39:05.294827 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:05.294792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:05.505788 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:05.505758 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mn26j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 20:39:05.506163 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:05.505822 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 20:39:05.521003 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:05.520966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:06.467101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.467068 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr"] Jul 16 20:39:06.470709 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.470686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.473557 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.473535 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n8bzs\"" Jul 16 20:39:06.485206 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.485180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr"] Jul 16 20:39:06.500156 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80aa256-e1ca-4412-8aa3-780d26014fff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwx7\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-kube-api-access-5wwx7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500314 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500476 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500476 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.500476 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.500441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.547118 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.547087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:06.601418 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601590 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601590 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601715 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80aa256-e1ca-4412-8aa3-780d26014fff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601715 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwx7\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-kube-api-access-5wwx7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601715 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601869 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601869 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601869 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.601869 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.602090 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.601880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.602222 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.602174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.602222 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.602228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.602715 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.602693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80aa256-e1ca-4412-8aa3-780d26014fff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.604411 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.604376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.604717 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.604700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.627546 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.627510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.629430 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.629403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwx7\" (UniqueName: \"kubernetes.io/projected/a80aa256-e1ca-4412-8aa3-780d26014fff-kube-api-access-5wwx7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45cpmr\" (UID: \"a80aa256-e1ca-4412-8aa3-780d26014fff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.781849 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.781808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:06.910942 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.910915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:06.943422 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:06.943399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr"] Jul 16 20:39:06.945715 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:39:06.945690 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80aa256_e1ca_4412_8aa3_780d26014fff.slice/crio-cd1ee1fda3ae22cb22bfcee81d2e2e315d881b6cbb1573e45cab67e5612d3b35 WatchSource:0}: Error finding container cd1ee1fda3ae22cb22bfcee81d2e2e315d881b6cbb1573e45cab67e5612d3b35: Status 404 returned error can't find the container with id cd1ee1fda3ae22cb22bfcee81d2e2e315d881b6cbb1573e45cab67e5612d3b35 Jul 16 20:39:07.007127 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:07.007088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:07.517233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:07.517191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" event={"ID":"a80aa256-e1ca-4412-8aa3-780d26014fff","Type":"ContainerStarted","Data":"cd1ee1fda3ae22cb22bfcee81d2e2e315d881b6cbb1573e45cab67e5612d3b35"} Jul 16 20:39:08.130095 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.130055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:08.274685 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.274652 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-krhtj"] Jul 16 20:39:08.278809 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.278780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.293466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.293439 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 16 20:39:08.293682 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.293661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-998pp\"" Jul 16 20:39:08.320093 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.320066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95fj\" (UniqueName: \"kubernetes.io/projected/c684a01c-63a0-4c84-b5f6-56dd8a199320-kube-api-access-q95fj\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.320283 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.320259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.332277 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.332207 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-krhtj"] Jul 16 20:39:08.372075 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.372042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:08.421911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.421829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.421911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.421877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95fj\" (UniqueName: \"kubernetes.io/projected/c684a01c-63a0-4c84-b5f6-56dd8a199320-kube-api-access-q95fj\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.422115 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:39:08.421981 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 16 20:39:08.422115 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:39:08.422052 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert podName:c684a01c-63a0-4c84-b5f6-56dd8a199320 nodeName:}" failed. No retries permitted until 2026-07-16 20:39:08.922031254 +0000 UTC m=+347.423111966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert") pod "kserve-controller-manager-d6f4b65b8-krhtj" (UID: "c684a01c-63a0-4c84-b5f6-56dd8a199320") : secret "kserve-webhook-server-cert" not found Jul 16 20:39:08.446209 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.446169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95fj\" (UniqueName: \"kubernetes.io/projected/c684a01c-63a0-4c84-b5f6-56dd8a199320-kube-api-access-q95fj\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.506832 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.505873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:39:08.547057 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.547016 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:39:08.547256 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.547230 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerName="discovery" containerID="cri-o://2fca0b2f06dc8d29eab0482cf556fbf3e4b1e39efcd82cd07a439579fd1d51b0" gracePeriod=30 Jul 16 20:39:08.586138 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.586095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:08.792854 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.792822 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:08.927199 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.927153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:08.929989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:08.929965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c684a01c-63a0-4c84-b5f6-56dd8a199320-cert\") pod \"kserve-controller-manager-d6f4b65b8-krhtj\" (UID: \"c684a01c-63a0-4c84-b5f6-56dd8a199320\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:09.193783 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.193691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:09.533633 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.533457 2564 generic.go:356] "Generic (PLEG): container finished" podID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerID="2fca0b2f06dc8d29eab0482cf556fbf3e4b1e39efcd82cd07a439579fd1d51b0" exitCode=0 Jul 16 20:39:09.533633 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.533558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" event={"ID":"1565d4e9-2767-43cb-96bc-766cb24fe6fe","Type":"ContainerDied","Data":"2fca0b2f06dc8d29eab0482cf556fbf3e4b1e39efcd82cd07a439579fd1d51b0"} Jul 16 20:39:09.543288 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.543226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:09.551284 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.551248 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:39:09.551400 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.551356 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:39:09.551400 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.551393 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:39:09.624451 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.624426 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:39:09.646902 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.646879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-krhtj"] Jul 16 20:39:09.648819 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:39:09.648793 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc684a01c_63a0_4c84_b5f6_56dd8a199320.slice/crio-ba1172f922da3e2d8591de4447fe8497427fbd26c89c94079d32dac7c5c63254 WatchSource:0}: Error finding container ba1172f922da3e2d8591de4447fe8497427fbd26c89c94079d32dac7c5c63254: Status 404 returned error can't find the container with id ba1172f922da3e2d8591de4447fe8497427fbd26c89c94079d32dac7c5c63254 Jul 16 20:39:09.660849 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.660804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:09.734616 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734616 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734610 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734706 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7c5z\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.734828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.734783 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token\") pod \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\" (UID: \"1565d4e9-2767-43cb-96bc-766cb24fe6fe\") " Jul 16 20:39:09.735074 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.735030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 20:39:09.737399 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737372 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:39:09.737540 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737511 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts" (OuterVolumeSpecName: "cacerts") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 20:39:09.737607 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 20:39:09.737646 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 20:39:09.737646 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737588 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z" (OuterVolumeSpecName: "kube-api-access-b7c5z") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "kube-api-access-b7c5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:39:09.737710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.737694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs" (OuterVolumeSpecName: "local-certs") pod "1565d4e9-2767-43cb-96bc-766cb24fe6fe" (UID: "1565d4e9-2767-43cb-96bc-766cb24fe6fe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 20:39:09.835703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835661 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-kubeconfig\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835695 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-ca-configmap\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835704 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-csr-dns-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835713 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7c5z\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-kube-api-access-b7c5z\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835722 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1565d4e9-2767-43cb-96bc-766cb24fe6fe-local-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835731 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1565d4e9-2767-43cb-96bc-766cb24fe6fe-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.835959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.835739 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1565d4e9-2767-43cb-96bc-766cb24fe6fe-cacerts\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:09.860969 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:09.860942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:10.540660 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.540622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" event={"ID":"a80aa256-e1ca-4412-8aa3-780d26014fff","Type":"ContainerStarted","Data":"8352ccd754b8a28ffb04b6fa2ed8a30e464315d63caabf267b5666fe4fc74674"} Jul 16 20:39:10.542105 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.542075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" event={"ID":"c684a01c-63a0-4c84-b5f6-56dd8a199320","Type":"ContainerStarted","Data":"ba1172f922da3e2d8591de4447fe8497427fbd26c89c94079d32dac7c5c63254"} Jul 16 20:39:10.543777 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.543748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" event={"ID":"1565d4e9-2767-43cb-96bc-766cb24fe6fe","Type":"ContainerDied","Data":"f3a2b884573562866b8ee01b0bc2e7fc0d0ac309e722e38a66607a0a50943b80"} Jul 16 20:39:10.543910 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.543798 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mn26j" Jul 16 20:39:10.543910 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.543813 2564 scope.go:117] "RemoveContainer" containerID="2fca0b2f06dc8d29eab0482cf556fbf3e4b1e39efcd82cd07a439579fd1d51b0" Jul 16 20:39:10.567221 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.567169 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" podStartSLOduration=1.964026335 podStartE2EDuration="4.567150136s" totalImagesPullingTime="2.603123801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:10.565267386 +0000 UTC m=+349.066348113" watchObservedRunningTime="2026-07-16 20:39:10.567150136 +0000 UTC m=+349.068230862" Jul 16 20:39:10.579624 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.579588 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:39:10.585589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.585559 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mn26j"] Jul 16 20:39:10.782500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.782459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:10.788491 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:10.788455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:11.550447 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:11.550410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:11.552002 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:11.551974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45cpmr" Jul 16 20:39:12.038702 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:12.038664 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" path="/var/lib/kubelet/pods/1565d4e9-2767-43cb-96bc-766cb24fe6fe/volumes" Jul 16 20:39:12.488696 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:12.488665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-x5l4z" Jul 16 20:39:12.490661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:12.490635 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-j2k9j" Jul 16 20:39:12.557019 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:12.556985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" event={"ID":"c684a01c-63a0-4c84-b5f6-56dd8a199320","Type":"ContainerStarted","Data":"161459397f865873f37eec0262ca24357d87cfbaf632142a001f334b3f0e7042"} Jul 16 20:39:12.586688 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:12.586634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" podStartSLOduration=2.430283012 podStartE2EDuration="4.586613125s" totalImagesPullingTime="2.156330113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:12.584392988 +0000 UTC m=+351.085473716" watchObservedRunningTime="2026-07-16 20:39:12.586613125 +0000 UTC m=+351.087693852" Jul 16 20:39:13.561771 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:13.561740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:44.571987 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:44.571951 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-krhtj" Jul 16 20:39:55.944648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:55.944604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:56.050841 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.050802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:56.081652 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.081618 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:39:56.081956 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.081943 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerName="discovery" Jul 16 20:39:56.081998 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.081959 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerName="discovery" Jul 16 20:39:56.082035 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.082029 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1565d4e9-2767-43cb-96bc-766cb24fe6fe" containerName="discovery" Jul 16 20:39:56.087169 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.087146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:56.090385 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.090362 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 20:39:56.090542 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.090365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 20:39:56.090542 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.090412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gj9fh\"" Jul 16 20:39:56.101458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.101432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:39:56.149937 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.149895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:56.152212 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.152187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fpz\" (UniqueName: \"kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz\") pod \"kuadrant-operator-catalog-tbffb\" (UID: \"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:56.253571 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.253534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fpz\" (UniqueName: \"kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz\") pod \"kuadrant-operator-catalog-tbffb\" (UID: \"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:56.261944 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.261912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fpz\" (UniqueName: \"kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz\") pod \"kuadrant-operator-catalog-tbffb\" (UID: \"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab\") " pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:56.399247 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.399213 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:56.433072 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.433041 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:39:56.534448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.534421 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:39:56.537135 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:39:56.537104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0808f446_8d33_4c4e_ad3c_6d6f4888f4ab.slice/crio-1dd939eae0c25f002e5808dc0d32f2e59e463461853314582cc3925d89b8cd13 WatchSource:0}: Error finding container 1dd939eae0c25f002e5808dc0d32f2e59e463461853314582cc3925d89b8cd13: Status 404 returned error can't find the container with id 1dd939eae0c25f002e5808dc0d32f2e59e463461853314582cc3925d89b8cd13 Jul 16 20:39:56.642191 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.642155 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t9558"] Jul 16 20:39:56.644999 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.644982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:39:56.652327 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.652277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t9558"] Jul 16 20:39:56.746061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.746023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" event={"ID":"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab","Type":"ContainerStarted","Data":"1dd939eae0c25f002e5808dc0d32f2e59e463461853314582cc3925d89b8cd13"} Jul 16 20:39:56.758612 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.758579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54tl\" (UniqueName: \"kubernetes.io/projected/7dbd4fa1-91ab-436d-8dff-2bbb32aa171f-kube-api-access-b54tl\") pod \"kuadrant-operator-catalog-t9558\" (UID: \"7dbd4fa1-91ab-436d-8dff-2bbb32aa171f\") " pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:39:56.860017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.859921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54tl\" (UniqueName: \"kubernetes.io/projected/7dbd4fa1-91ab-436d-8dff-2bbb32aa171f-kube-api-access-b54tl\") pod \"kuadrant-operator-catalog-t9558\" (UID: \"7dbd4fa1-91ab-436d-8dff-2bbb32aa171f\") " pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:39:56.863805 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.863778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:56.867998 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.867978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54tl\" (UniqueName: \"kubernetes.io/projected/7dbd4fa1-91ab-436d-8dff-2bbb32aa171f-kube-api-access-b54tl\") pod \"kuadrant-operator-catalog-t9558\" (UID: \"7dbd4fa1-91ab-436d-8dff-2bbb32aa171f\") " pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:39:56.957532 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:56.957494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:39:57.072899 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:57.072873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:39:57.148257 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:57.148236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t9558"] Jul 16 20:39:57.154379 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:39:57.154350 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbd4fa1_91ab_436d_8dff_2bbb32aa171f.slice/crio-31f5cd1552baac06164c6a34b7fced54943220c69990f37179f06c2ef2fd94b8 WatchSource:0}: Error finding container 31f5cd1552baac06164c6a34b7fced54943220c69990f37179f06c2ef2fd94b8: Status 404 returned error can't find the container with id 31f5cd1552baac06164c6a34b7fced54943220c69990f37179f06c2ef2fd94b8 Jul 16 20:39:57.752662 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:57.752621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t9558" event={"ID":"7dbd4fa1-91ab-436d-8dff-2bbb32aa171f","Type":"ContainerStarted","Data":"31f5cd1552baac06164c6a34b7fced54943220c69990f37179f06c2ef2fd94b8"} Jul 16 20:39:58.758379 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.758327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" event={"ID":"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab","Type":"ContainerStarted","Data":"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a"} Jul 16 20:39:58.758885 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.758407 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" podUID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" containerName="registry-server" containerID="cri-o://8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a" gracePeriod=2 Jul 16 20:39:58.759825 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.759799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t9558" event={"ID":"7dbd4fa1-91ab-436d-8dff-2bbb32aa171f","Type":"ContainerStarted","Data":"955e702c86fe77cfa71b9ea019e9fa7a4effa3e751ad3ec59c295a1f202e6187"} Jul 16 20:39:58.774189 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.774145 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" podStartSLOduration=0.893632536 podStartE2EDuration="2.774120203s" totalImagesPullingTime="1.880487667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:58.773271997 +0000 UTC m=+397.274352726" watchObservedRunningTime="2026-07-16 20:39:58.774120203 +0000 UTC m=+397.275200930" Jul 16 20:39:58.788505 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.788451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-t9558" podStartSLOduration=1.524247036 podStartE2EDuration="2.788431617s" totalImagesPullingTime="1.264184581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:58.788053574 +0000 UTC m=+397.289134300" watchObservedRunningTime="2026-07-16 20:39:58.788431617 +0000 UTC m=+397.289512344" Jul 16 20:39:58.996886 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:58.996861 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:59.078984 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.078892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79fpz\" (UniqueName: \"kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz\") pod \"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab\" (UID: \"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab\") " Jul 16 20:39:59.081183 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.081161 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz" (OuterVolumeSpecName: "kube-api-access-79fpz") pod "0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" (UID: "0808f446-8d33-4c4e-ad3c-6d6f4888f4ab"). InnerVolumeSpecName "kube-api-access-79fpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:39:59.179622 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.179586 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79fpz\" (UniqueName: \"kubernetes.io/projected/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab-kube-api-access-79fpz\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:39:59.765641 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.765611 2564 generic.go:356] "Generic (PLEG): container finished" podID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" containerID="8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a" exitCode=0 Jul 16 20:39:59.766017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.765659 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" Jul 16 20:39:59.766017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.765704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" event={"ID":"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab","Type":"ContainerDied","Data":"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a"} Jul 16 20:39:59.766017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.765748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbffb" event={"ID":"0808f446-8d33-4c4e-ad3c-6d6f4888f4ab","Type":"ContainerDied","Data":"1dd939eae0c25f002e5808dc0d32f2e59e463461853314582cc3925d89b8cd13"} Jul 16 20:39:59.766017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.765771 2564 scope.go:117] "RemoveContainer" containerID="8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a" Jul 16 20:39:59.774658 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.774641 2564 scope.go:117] "RemoveContainer" containerID="8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a" Jul 16 20:39:59.774918 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:39:59.774898 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a\": container with ID starting with 8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a not found: ID does not exist" containerID="8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a" Jul 16 20:39:59.774962 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.774928 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a"} err="failed to get container status \"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a\": rpc error: code = NotFound desc = could not find container \"8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a\": container with ID starting with 8f7f26b27b97afac3e5dc97f97b62a6eca8a2c364e25f17c0d6d6b861f49806a not found: ID does not exist" Jul 16 20:39:59.786745 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.786712 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:39:59.791433 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:39:59.791407 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbffb"] Jul 16 20:40:00.038988 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:00.038913 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" path="/var/lib/kubelet/pods/0808f446-8d33-4c4e-ad3c-6d6f4888f4ab/volumes" Jul 16 20:40:06.958398 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:06.958345 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:40:06.958828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:06.958428 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:40:06.981090 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:06.981063 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:40:07.825075 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:07.825044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-t9558" Jul 16 20:40:24.305213 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:24.305175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:27.785264 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:27.785235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:29.490758 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.490716 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8"] Jul 16 20:40:29.491257 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.491236 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" containerName="registry-server" Jul 16 20:40:29.491363 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.491260 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" containerName="registry-server" Jul 16 20:40:29.491424 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.491365 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0808f446-8d33-4c4e-ad3c-6d6f4888f4ab" containerName="registry-server" Jul 16 20:40:29.497847 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.497822 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.506356 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.506333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mb4tc\"" Jul 16 20:40:29.516381 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.516353 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8"] Jul 16 20:40:29.529217 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.529185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfstg\" (UniqueName: \"kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.529388 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.529223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.630050 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.630009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.630262 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.630161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfstg\" (UniqueName: \"kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.630465 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.630442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.646222 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.646190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfstg\" (UniqueName: \"kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.783144 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.783063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:29.808180 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.808142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:29.965341 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:29.965315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8"] Jul 16 20:40:29.967832 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:29.967806 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0573f288_788e_4b0b_a825_95dd7f658d11.slice/crio-334c14b607a5bc05671afbed838277ef72fb15c3b8e207112e7ef978f1e9de75 WatchSource:0}: Error finding container 334c14b607a5bc05671afbed838277ef72fb15c3b8e207112e7ef978f1e9de75: Status 404 returned error can't find the container with id 334c14b607a5bc05671afbed838277ef72fb15c3b8e207112e7ef978f1e9de75 Jul 16 20:40:30.908640 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:30.908602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" event={"ID":"0573f288-788e-4b0b-a825-95dd7f658d11","Type":"ContainerStarted","Data":"334c14b607a5bc05671afbed838277ef72fb15c3b8e207112e7ef978f1e9de75"} Jul 16 20:40:31.003287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.003250 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf"] Jul 16 20:40:31.006586 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.006568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:31.009689 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.009664 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vwb6t\"" Jul 16 20:40:31.009689 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.009677 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 20:40:31.030925 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.030889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf"] Jul 16 20:40:31.043370 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.043338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wk4z\" (UniqueName: \"kubernetes.io/projected/e4dea57f-2641-4933-97d5-c86f67aaef7b-kube-api-access-4wk4z\") pod \"dns-operator-controller-manager-864dd78c4c-twhkf\" (UID: \"e4dea57f-2641-4933-97d5-c86f67aaef7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:31.144812 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.144778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wk4z\" (UniqueName: \"kubernetes.io/projected/e4dea57f-2641-4933-97d5-c86f67aaef7b-kube-api-access-4wk4z\") pod \"dns-operator-controller-manager-864dd78c4c-twhkf\" (UID: \"e4dea57f-2641-4933-97d5-c86f67aaef7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:31.160548 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.160471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wk4z\" (UniqueName: \"kubernetes.io/projected/e4dea57f-2641-4933-97d5-c86f67aaef7b-kube-api-access-4wk4z\") pod \"dns-operator-controller-manager-864dd78c4c-twhkf\" (UID: \"e4dea57f-2641-4933-97d5-c86f67aaef7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:31.317088 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.317053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:31.388249 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.388218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:31.499367 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.499340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf"] Jul 16 20:40:31.501506 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:31.501468 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dea57f_2641_4933_97d5_c86f67aaef7b.slice/crio-2c45ccb94c1f132ca58e5ce527553c292dac1ce203e0e3884869cb43eea6d230 WatchSource:0}: Error finding container 2c45ccb94c1f132ca58e5ce527553c292dac1ce203e0e3884869cb43eea6d230: Status 404 returned error can't find the container with id 2c45ccb94c1f132ca58e5ce527553c292dac1ce203e0e3884869cb43eea6d230 Jul 16 20:40:31.746099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.746066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:31.865539 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.865504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:31.914448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:31.914413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" event={"ID":"e4dea57f-2641-4933-97d5-c86f67aaef7b","Type":"ContainerStarted","Data":"2c45ccb94c1f132ca58e5ce527553c292dac1ce203e0e3884869cb43eea6d230"} Jul 16 20:40:34.276864 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.276826 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8"] Jul 16 20:40:34.280514 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.280493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:34.283225 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.283204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-prg7n\"" Jul 16 20:40:34.292174 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.292149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8"] Jul 16 20:40:34.377155 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.377115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5kl\" (UniqueName: \"kubernetes.io/projected/9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e-kube-api-access-qq5kl\") pod \"authorino-operator-75cc8d5cb7-zhzx8\" (UID: \"9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:34.477743 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.477705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5kl\" (UniqueName: \"kubernetes.io/projected/9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e-kube-api-access-qq5kl\") pod \"authorino-operator-75cc8d5cb7-zhzx8\" (UID: \"9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:34.488009 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.487976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5kl\" (UniqueName: \"kubernetes.io/projected/9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e-kube-api-access-qq5kl\") pod \"authorino-operator-75cc8d5cb7-zhzx8\" (UID: \"9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:34.593695 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.593622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:34.768828 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.768802 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8"] Jul 16 20:40:34.771511 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:34.771486 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4c3b83_9b6e_44cc_b9f2_16c3475dcf2e.slice/crio-db953a4311c99e03cf7ccba6bba474cb6e5e2fb29b05e0d80bfdd3375d72ee1b WatchSource:0}: Error finding container db953a4311c99e03cf7ccba6bba474cb6e5e2fb29b05e0d80bfdd3375d72ee1b: Status 404 returned error can't find the container with id db953a4311c99e03cf7ccba6bba474cb6e5e2fb29b05e0d80bfdd3375d72ee1b Jul 16 20:40:34.906601 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.906519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:34.931014 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.930975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" event={"ID":"0573f288-788e-4b0b-a825-95dd7f658d11","Type":"ContainerStarted","Data":"36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a"} Jul 16 20:40:34.931211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.931093 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:34.932373 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.932347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" event={"ID":"9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e","Type":"ContainerStarted","Data":"db953a4311c99e03cf7ccba6bba474cb6e5e2fb29b05e0d80bfdd3375d72ee1b"} Jul 16 20:40:34.933787 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.933767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" event={"ID":"e4dea57f-2641-4933-97d5-c86f67aaef7b","Type":"ContainerStarted","Data":"95f8e1a3006cb796204e25e3dc11f5306fda1c55a7f19ba42fc9fe73ff2218ae"} Jul 16 20:40:34.933900 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.933816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:34.951109 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.951064 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" podStartSLOduration=1.29000793 podStartE2EDuration="5.951048706s" totalImagesPullingTime="4.661040776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:34.949397774 +0000 UTC m=+433.450478502" watchObservedRunningTime="2026-07-16 20:40:34.951048706 +0000 UTC m=+433.452129432" Jul 16 20:40:34.969175 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:34.969123 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" podStartSLOduration=1.854601666 podStartE2EDuration="4.969105995s" totalImagesPullingTime="3.114504329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:34.968982231 +0000 UTC m=+433.470062960" watchObservedRunningTime="2026-07-16 20:40:34.969105995 +0000 UTC m=+433.470186722" Jul 16 20:40:35.193496 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:35.193410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:36.927341 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:36.927315 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4"] Jul 16 20:40:36.930814 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:36.930792 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:36.933427 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:36.933408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-49lxv\"" Jul 16 20:40:36.946176 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:36.946145 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4"] Jul 16 20:40:37.001149 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.001120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlszn\" (UniqueName: \"kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn\") pod \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" (UID: \"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:37.101643 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.101608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlszn\" (UniqueName: \"kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn\") pod \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" (UID: \"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:37.113894 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.113863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlszn\" (UniqueName: \"kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn\") pod \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" (UID: \"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:37.241214 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.241179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:37.384374 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.384344 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4"] Jul 16 20:40:37.386253 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:37.386228 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7e4b51_6c1e_42e4_9dd3_2008f2af2959.slice/crio-6c2664d01cee6e76914fd02b6c718bc29479aaa0ec7e09a23265ee96033ab8de WatchSource:0}: Error finding container 6c2664d01cee6e76914fd02b6c718bc29479aaa0ec7e09a23265ee96033ab8de: Status 404 returned error can't find the container with id 6c2664d01cee6e76914fd02b6c718bc29479aaa0ec7e09a23265ee96033ab8de Jul 16 20:40:37.798461 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.798421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:37.956838 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.956801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" event={"ID":"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959","Type":"ContainerStarted","Data":"6c2664d01cee6e76914fd02b6c718bc29479aaa0ec7e09a23265ee96033ab8de"} Jul 16 20:40:37.957359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.957334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:37.958926 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.958898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" event={"ID":"9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e","Type":"ContainerStarted","Data":"0cdf9b2773ec9d5c9c520eb1a7953a2d58730087168f3ad635308a02d5c9958b"} Jul 16 20:40:37.959170 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.959151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:37.980061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:37.980000 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" podStartSLOduration=1.8686633929999998 podStartE2EDuration="3.979982234s" totalImagesPullingTime="2.111318841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:37.979168241 +0000 UTC m=+436.480248979" watchObservedRunningTime="2026-07-16 20:40:37.979982234 +0000 UTC m=+436.481063027" Jul 16 20:40:38.168686 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:38.168609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:38.964985 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:38.964893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" event={"ID":"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959","Type":"ContainerStarted","Data":"d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55"} Jul 16 20:40:38.965338 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:38.965029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:38.985347 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:38.985294 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" podStartSLOduration=1.769071951 podStartE2EDuration="2.985281521s" totalImagesPullingTime="1.21620957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:38.983714913 +0000 UTC m=+437.484795642" watchObservedRunningTime="2026-07-16 20:40:38.985281521 +0000 UTC m=+437.486362246" Jul 16 20:40:39.513053 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:39.513016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:39.590718 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:39.590679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:45.941788 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:45.941708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:45.941788 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:45.941782 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twhkf" Jul 16 20:40:47.265876 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.265849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:47.410990 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.410959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:47.925673 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.925623 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8"] Jul 16 20:40:47.925867 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.925811 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" containerName="manager" containerID="cri-o://36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a" gracePeriod=2 Jul 16 20:40:47.940887 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.940858 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8"] Jul 16 20:40:47.965229 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.965197 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4"] Jul 16 20:40:47.965486 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.965457 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" containerName="manager" containerID="cri-o://d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55" gracePeriod=2 Jul 16 20:40:47.967350 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.967323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:47.976883 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:47.976848 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0573f288-788e-4b0b-a825-95dd7f658d11\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\": pods \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:47.980037 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:47.979461 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:47.981206 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.981182 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4"] Jul 16 20:40:47.982081 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:47.982056 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:47.983500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.983476 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:40:47.983966 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.983942 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" containerName="manager" Jul 16 20:40:47.984271 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.984085 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" containerName="manager" Jul 16 20:40:47.984271 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.984140 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" containerName="manager" Jul 16 20:40:47.984271 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.984150 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" containerName="manager" Jul 16 20:40:47.984271 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.984255 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" containerName="manager" Jul 16 20:40:47.984271 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.984268 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" containerName="manager" Jul 16 20:40:47.987160 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:47.987144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.008289 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.008253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:40:48.015442 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.015416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq"] Jul 16 20:40:48.018855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.018827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:48.031267 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.031244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq"] Jul 16 20:40:48.042907 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:48.042877 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vqcv8\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:48.049455 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:48.049428 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:48.098785 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.098756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcsp\" (UniqueName: \"kubernetes.io/projected/fe0eba06-dc02-4011-832f-b0108fd2458a-kube-api-access-clcsp\") pod \"limitador-operator-controller-manager-859d7b78bd-pb8kq\" (UID: \"fe0eba06-dc02-4011-832f-b0108fd2458a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:48.098941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.098797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjj4\" (UniqueName: \"kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.098941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.098910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.199960 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.199579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcsp\" (UniqueName: \"kubernetes.io/projected/fe0eba06-dc02-4011-832f-b0108fd2458a-kube-api-access-clcsp\") pod \"limitador-operator-controller-manager-859d7b78bd-pb8kq\" (UID: \"fe0eba06-dc02-4011-832f-b0108fd2458a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:48.199960 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.199628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjj4\" (UniqueName: \"kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.199960 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.199705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.200174 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.200123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.221101 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.221060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjj4\" (UniqueName: \"kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vqdq4\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.221612 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.221589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcsp\" (UniqueName: \"kubernetes.io/projected/fe0eba06-dc02-4011-832f-b0108fd2458a-kube-api-access-clcsp\") pod \"limitador-operator-controller-manager-859d7b78bd-pb8kq\" (UID: \"fe0eba06-dc02-4011-832f-b0108fd2458a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:48.221727 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.221624 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:48.224855 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.224840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:48.300799 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.300766 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlszn\" (UniqueName: \"kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn\") pod \"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959\" (UID: \"ab7e4b51-6c1e-42e4-9dd3-2008f2af2959\") " Jul 16 20:40:48.301155 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.300834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume\") pod \"0573f288-788e-4b0b-a825-95dd7f658d11\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " Jul 16 20:40:48.301155 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.300861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfstg\" (UniqueName: \"kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg\") pod \"0573f288-788e-4b0b-a825-95dd7f658d11\" (UID: \"0573f288-788e-4b0b-a825-95dd7f658d11\") " Jul 16 20:40:48.301384 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.301362 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0573f288-788e-4b0b-a825-95dd7f658d11" (UID: "0573f288-788e-4b0b-a825-95dd7f658d11"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 20:40:48.303011 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.302989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn" (OuterVolumeSpecName: "kube-api-access-vlszn") pod "ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" (UID: "ab7e4b51-6c1e-42e4-9dd3-2008f2af2959"). InnerVolumeSpecName "kube-api-access-vlszn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:40:48.303071 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.302995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg" (OuterVolumeSpecName: "kube-api-access-mfstg") pod "0573f288-788e-4b0b-a825-95dd7f658d11" (UID: "0573f288-788e-4b0b-a825-95dd7f658d11"). InnerVolumeSpecName "kube-api-access-mfstg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:40:48.348897 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.348862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:48.354915 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.354894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:48.402428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.402398 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0573f288-788e-4b0b-a825-95dd7f658d11-extensions-socket-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:40:48.402547 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.402429 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfstg\" (UniqueName: \"kubernetes.io/projected/0573f288-788e-4b0b-a825-95dd7f658d11-kube-api-access-mfstg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:40:48.402547 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.402445 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlszn\" (UniqueName: \"kubernetes.io/projected/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959-kube-api-access-vlszn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:40:48.514757 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:48.514722 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b32600_2e9c_4e28_a893_0031a7e2767d.slice/crio-cbbc85b2511d409e43b73ee37815ef630216017b5bdb915a5c92b447bec4a292 WatchSource:0}: Error finding container cbbc85b2511d409e43b73ee37815ef630216017b5bdb915a5c92b447bec4a292: Status 404 returned error can't find the container with id cbbc85b2511d409e43b73ee37815ef630216017b5bdb915a5c92b447bec4a292 Jul 16 20:40:48.519049 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.518914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:40:48.542047 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.541999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq"] Jul 16 20:40:48.550436 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:48.550401 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0eba06_dc02_4011_832f_b0108fd2458a.slice/crio-fa35768325ace44c47530155af9c588996f30c602ab88fbfb255f280054e62b8 WatchSource:0}: Error finding container fa35768325ace44c47530155af9c588996f30c602ab88fbfb255f280054e62b8: Status 404 returned error can't find the container with id fa35768325ace44c47530155af9c588996f30c602ab88fbfb255f280054e62b8 Jul 16 20:40:48.967267 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:48.967237 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zhzx8" Jul 16 20:40:49.013111 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.013081 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" containerID="d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55" exitCode=0 Jul 16 20:40:49.013296 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.013142 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:49.013296 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.013161 2564 scope.go:117] "RemoveContainer" containerID="d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55" Jul 16 20:40:49.015362 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.015334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" event={"ID":"fe0eba06-dc02-4011-832f-b0108fd2458a","Type":"ContainerStarted","Data":"bc9dd9040060b6f077990a79f29806965147769a8b68f01aa67b52d07fb362c0"} Jul 16 20:40:49.015842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.015783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" event={"ID":"fe0eba06-dc02-4011-832f-b0108fd2458a","Type":"ContainerStarted","Data":"fa35768325ace44c47530155af9c588996f30c602ab88fbfb255f280054e62b8"} Jul 16 20:40:49.015842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.015830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:40:49.017185 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.017161 2564 generic.go:356] "Generic (PLEG): container finished" podID="0573f288-788e-4b0b-a825-95dd7f658d11" containerID="36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a" exitCode=0 Jul 16 20:40:49.017339 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.017218 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqcv8" Jul 16 20:40:49.020318 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.020275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" event={"ID":"b1b32600-2e9c-4e28-a893-0031a7e2767d","Type":"ContainerStarted","Data":"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea"} Jul 16 20:40:49.020410 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.020352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" event={"ID":"b1b32600-2e9c-4e28-a893-0031a7e2767d","Type":"ContainerStarted","Data":"cbbc85b2511d409e43b73ee37815ef630216017b5bdb915a5c92b447bec4a292"} Jul 16 20:40:49.020410 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.020406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:40:49.023981 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.023964 2564 scope.go:117] "RemoveContainer" containerID="d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55" Jul 16 20:40:49.024277 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:49.024249 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55\": container with ID starting with d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55 not found: ID does not exist" containerID="d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55" Jul 16 20:40:49.024393 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.024288 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55"} err="failed to get container status \"d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55\": rpc error: code = NotFound desc = could not find container \"d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55\": container with ID starting with d472052ef9f1f4c510eba2656248f2d9f34725d7e1f08f23da8c299de6575b55 not found: ID does not exist" Jul 16 20:40:49.024393 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.024342 2564 scope.go:117] "RemoveContainer" containerID="36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a" Jul 16 20:40:49.035479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.034170 2564 scope.go:117] "RemoveContainer" containerID="36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a" Jul 16 20:40:49.036256 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:49.036224 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a\": container with ID starting with 36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a not found: ID does not exist" containerID="36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a" Jul 16 20:40:49.036397 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.036264 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a"} err="failed to get container status \"36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a\": rpc error: code = NotFound desc = could not find container \"36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a\": container with ID starting with 36ee7887fbe04a6d0af62fa8335a573ebb71c07955bea0b1a01d1b03ba2c782a not found: ID does not exist" Jul 16 20:40:49.047540 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.047473 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" podStartSLOduration=2.047450434 podStartE2EDuration="2.047450434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:49.044737197 +0000 UTC m=+447.545817924" watchObservedRunningTime="2026-07-16 20:40:49.047450434 +0000 UTC m=+447.548531170" Jul 16 20:40:49.077894 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:49.077848 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:49.078272 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.078238 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" podStartSLOduration=2.078225421 podStartE2EDuration="2.078225421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:49.075355115 +0000 UTC m=+447.576435841" watchObservedRunningTime="2026-07-16 20:40:49.078225421 +0000 UTC m=+447.579306219" Jul 16 20:40:49.464401 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.464361 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:40:49.469167 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.469146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.471881 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:49.471821 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:49.491556 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.491523 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:40:49.614536 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.614497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dhb\" (UniqueName: \"kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.614723 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.614625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.715949 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.715846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.715949 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.715911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dhb\" (UniqueName: \"kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.716386 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.716365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.732933 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.732904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dhb\" (UniqueName: \"kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb\") pod \"kuadrant-operator-controller-manager-86878c769f-8pnr4\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.780106 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.780071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:49.939987 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:49.939948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:40:49.953117 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:40:49.953088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4e08b6_f3a7_40bd_9564_3c3ff0d246dd.slice/crio-72b7817cedcaf02a8848f98cf4b1cd8c9f6744926858b7c1d373d11ed72f60db WatchSource:0}: Error finding container 72b7817cedcaf02a8848f98cf4b1cd8c9f6744926858b7c1d373d11ed72f60db: Status 404 returned error can't find the container with id 72b7817cedcaf02a8848f98cf4b1cd8c9f6744926858b7c1d373d11ed72f60db Jul 16 20:40:50.030009 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:50.029978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" event={"ID":"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd","Type":"ContainerStarted","Data":"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e"} Jul 16 20:40:50.030166 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:50.030032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" event={"ID":"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd","Type":"ContainerStarted","Data":"72b7817cedcaf02a8848f98cf4b1cd8c9f6744926858b7c1d373d11ed72f60db"} Jul 16 20:40:50.039888 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:50.039852 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0573f288-788e-4b0b-a825-95dd7f658d11" path="/var/lib/kubelet/pods/0573f288-788e-4b0b-a825-95dd7f658d11/volumes" Jul 16 20:40:50.040160 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:50.040147 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" path="/var/lib/kubelet/pods/ab7e4b51-6c1e-42e4-9dd3-2008f2af2959/volumes" Jul 16 20:40:50.066200 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:50.066147 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" podStartSLOduration=1.066131092 podStartE2EDuration="1.066131092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:50.064903636 +0000 UTC m=+448.565984353" watchObservedRunningTime="2026-07-16 20:40:50.066131092 +0000 UTC m=+448.567211801" Jul 16 20:40:50.067502 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:40:50.067475 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vtnq4\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="ab7e4b51-6c1e-42e4-9dd3-2008f2af2959" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vtnq4" Jul 16 20:40:51.035134 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:51.035091 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:40:53.529054 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:53.528993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:53.631067 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:53.631030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:54.133127 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:54.133091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:54.231652 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:54.231618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:40:59.956210 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:40:59.956177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:00.030450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:00.030418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:00.032018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:00.031992 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:41:00.032146 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:00.032070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pb8kq" Jul 16 20:41:00.254703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:00.254656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:02.045028 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.045002 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:41:02.108875 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.103496 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:41:02.108875 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.103981 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" podUID="b1b32600-2e9c-4e28-a893-0031a7e2767d" containerName="manager" containerID="cri-o://cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea" gracePeriod=10 Jul 16 20:41:02.354492 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.354468 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:41:02.529904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.529865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjj4\" (UniqueName: \"kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4\") pod \"b1b32600-2e9c-4e28-a893-0031a7e2767d\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " Jul 16 20:41:02.529904 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.529902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume\") pod \"b1b32600-2e9c-4e28-a893-0031a7e2767d\" (UID: \"b1b32600-2e9c-4e28-a893-0031a7e2767d\") " Jul 16 20:41:02.530358 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.530295 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1b32600-2e9c-4e28-a893-0031a7e2767d" (UID: "b1b32600-2e9c-4e28-a893-0031a7e2767d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 20:41:02.532191 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.532170 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4" (OuterVolumeSpecName: "kube-api-access-6xjj4") pod "b1b32600-2e9c-4e28-a893-0031a7e2767d" (UID: "b1b32600-2e9c-4e28-a893-0031a7e2767d"). InnerVolumeSpecName "kube-api-access-6xjj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:41:02.631091 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.631000 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjj4\" (UniqueName: \"kubernetes.io/projected/b1b32600-2e9c-4e28-a893-0031a7e2767d-kube-api-access-6xjj4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:41:02.631091 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:02.631031 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1b32600-2e9c-4e28-a893-0031a7e2767d-extensions-socket-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:41:03.116229 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.116185 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b32600-2e9c-4e28-a893-0031a7e2767d" containerID="cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea" exitCode=0 Jul 16 20:41:03.116229 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.116231 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" Jul 16 20:41:03.116713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.116265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" event={"ID":"b1b32600-2e9c-4e28-a893-0031a7e2767d","Type":"ContainerDied","Data":"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea"} Jul 16 20:41:03.116713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.116355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4" event={"ID":"b1b32600-2e9c-4e28-a893-0031a7e2767d","Type":"ContainerDied","Data":"cbbc85b2511d409e43b73ee37815ef630216017b5bdb915a5c92b447bec4a292"} Jul 16 20:41:03.116713 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.116386 2564 scope.go:117] "RemoveContainer" containerID="cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea" Jul 16 20:41:03.126048 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.126031 2564 scope.go:117] "RemoveContainer" containerID="cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea" Jul 16 20:41:03.126359 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:41:03.126340 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea\": container with ID starting with cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea not found: ID does not exist" containerID="cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea" Jul 16 20:41:03.126412 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.126370 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea"} err="failed to get container status \"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea\": rpc error: code = NotFound desc = could not find container \"cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea\": container with ID starting with cb902818115442efb6e195d2f2ad3cf4949011c6fd2c8bd4fe55a8059f4cfdea not found: ID does not exist" Jul 16 20:41:03.146196 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.146154 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:41:03.148055 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.148031 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vqdq4"] Jul 16 20:41:03.416666 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.416591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:03.515190 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:03.515158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:04.044768 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:04.044731 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b32600-2e9c-4e28-a893-0031a7e2767d" path="/var/lib/kubelet/pods/b1b32600-2e9c-4e28-a893-0031a7e2767d/volumes" Jul 16 20:41:08.191932 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:08.191903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:08.299207 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:08.299176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:11.165839 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:11.165806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:11.297703 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:11.297634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:17.709734 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:17.709700 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:17.808679 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:17.808649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:18.473913 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.473877 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl"] Jul 16 20:41:18.474480 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.474463 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b32600-2e9c-4e28-a893-0031a7e2767d" containerName="manager" Jul 16 20:41:18.474532 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.474485 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b32600-2e9c-4e28-a893-0031a7e2767d" containerName="manager" Jul 16 20:41:18.474576 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.474565 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b32600-2e9c-4e28-a893-0031a7e2767d" containerName="manager" Jul 16 20:41:18.477039 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.477014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.479646 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.479625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9km4z\"" Jul 16 20:41:18.487980 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.487949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl"] Jul 16 20:41:18.498817 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.498772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:18.568450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncc8\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-kube-api-access-cncc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a616930-461e-40da-8b27-e6052c45ccc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a616930-461e-40da-8b27-e6052c45ccc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.568710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.569061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.568740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.629006 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.628965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:18.669492 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669492 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncc8\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-kube-api-access-cncc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a616930-461e-40da-8b27-e6052c45ccc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a616930-461e-40da-8b27-e6052c45ccc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.669762 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670018 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.669935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670258 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.670234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670440 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.670291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670440 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.670392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.670594 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.670575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a616930-461e-40da-8b27-e6052c45ccc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.672701 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.672676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a616930-461e-40da-8b27-e6052c45ccc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.672912 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.672895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a616930-461e-40da-8b27-e6052c45ccc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.677500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.677476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.677908 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.677885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncc8\" (UniqueName: \"kubernetes.io/projected/5a616930-461e-40da-8b27-e6052c45ccc0-kube-api-access-cncc8\") pod \"maas-default-gateway-openshift-default-56654dcf7-xfnzl\" (UID: \"5a616930-461e-40da-8b27-e6052c45ccc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.788529 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.788431 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:18.928970 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.928901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl"] Jul 16 20:41:18.931889 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:41:18.931861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a616930_461e_40da_8b27_e6052c45ccc0.slice/crio-caf3dc8e045c3341748decb934983dafca366f35d87cf7356595e488563f589a WatchSource:0}: Error finding container caf3dc8e045c3341748decb934983dafca366f35d87cf7356595e488563f589a: Status 404 returned error can't find the container with id caf3dc8e045c3341748decb934983dafca366f35d87cf7356595e488563f589a Jul 16 20:41:18.934008 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.933969 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:41:18.934087 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.934047 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:41:18.934087 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:18.934079 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:41:19.194432 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:19.194327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" event={"ID":"5a616930-461e-40da-8b27-e6052c45ccc0","Type":"ContainerStarted","Data":"21d0922ca901b36cf2a512937085b92c2546051a25b7ee3951c1ef45c0dc1aa1"} Jul 16 20:41:19.194432 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:19.194395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" event={"ID":"5a616930-461e-40da-8b27-e6052c45ccc0","Type":"ContainerStarted","Data":"caf3dc8e045c3341748decb934983dafca366f35d87cf7356595e488563f589a"} Jul 16 20:41:19.213386 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:19.213335 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" podStartSLOduration=1.213314334 podStartE2EDuration="1.213314334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:19.211447947 +0000 UTC m=+477.712528674" watchObservedRunningTime="2026-07-16 20:41:19.213314334 +0000 UTC m=+477.714395052" Jul 16 20:41:19.789133 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:19.789090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:19.794342 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:19.794297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:20.199752 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:20.199675 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:20.200849 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:20.200823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:41:21.027770 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.027733 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:21.030388 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.030360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:21.033112 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.033086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4pxz6\"" Jul 16 20:41:21.036998 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.036968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:21.096152 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.096120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hn6\" (UniqueName: \"kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6\") pod \"authorino-58cd955675-wwdsb\" (UID: \"57422607-8d15-40bb-b7a9-807caa60e9a3\") " pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:21.194359 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.194316 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:21.196833 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.196806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:21.197211 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.197186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hn6\" (UniqueName: \"kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6\") pod \"authorino-58cd955675-wwdsb\" (UID: \"57422607-8d15-40bb-b7a9-807caa60e9a3\") " pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:21.203162 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.203102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:21.208234 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.208159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hn6\" (UniqueName: \"kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6\") pod \"authorino-58cd955675-wwdsb\" (UID: \"57422607-8d15-40bb-b7a9-807caa60e9a3\") " pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:21.298040 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.297941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvcn\" (UniqueName: \"kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn\") pod \"authorino-75d556bd77-4d55p\" (UID: \"d7a3de31-99d3-4527-ab2f-45db3660e884\") " pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:21.342617 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.342580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:21.398589 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.398558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvcn\" (UniqueName: \"kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn\") pod \"authorino-75d556bd77-4d55p\" (UID: \"d7a3de31-99d3-4527-ab2f-45db3660e884\") " pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:21.407190 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.407154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvcn\" (UniqueName: \"kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn\") pod \"authorino-75d556bd77-4d55p\" (UID: \"d7a3de31-99d3-4527-ab2f-45db3660e884\") " pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:21.479846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.479816 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:21.482630 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:41:21.482590 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57422607_8d15_40bb_b7a9_807caa60e9a3.slice/crio-daa5ebf334b5da0931b8d6f454aea32df1d07bfc5175b366ae8d2c0d384e0609 WatchSource:0}: Error finding container daa5ebf334b5da0931b8d6f454aea32df1d07bfc5175b366ae8d2c0d384e0609: Status 404 returned error can't find the container with id daa5ebf334b5da0931b8d6f454aea32df1d07bfc5175b366ae8d2c0d384e0609 Jul 16 20:41:21.509999 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.509960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:21.647524 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.647500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:21.649085 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:41:21.649056 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a3de31_99d3_4527_ab2f_45db3660e884.slice/crio-dcc81b8e8cf56cc81a6049d5ce3bf9ac2db4b6a2f2d95877e971ac0a9932c69f WatchSource:0}: Error finding container dcc81b8e8cf56cc81a6049d5ce3bf9ac2db4b6a2f2d95877e971ac0a9932c69f: Status 404 returned error can't find the container with id dcc81b8e8cf56cc81a6049d5ce3bf9ac2db4b6a2f2d95877e971ac0a9932c69f Jul 16 20:41:21.898820 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.898731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:21.963401 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:21.963369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:22.216833 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:22.216731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4d55p" event={"ID":"d7a3de31-99d3-4527-ab2f-45db3660e884","Type":"ContainerStarted","Data":"dcc81b8e8cf56cc81a6049d5ce3bf9ac2db4b6a2f2d95877e971ac0a9932c69f"} Jul 16 20:41:22.219353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:22.219318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wwdsb" event={"ID":"57422607-8d15-40bb-b7a9-807caa60e9a3","Type":"ContainerStarted","Data":"daa5ebf334b5da0931b8d6f454aea32df1d07bfc5175b366ae8d2c0d384e0609"} Jul 16 20:41:24.756236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:24.756186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:24.813852 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:24.813818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:25.244022 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:25.243989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4d55p" event={"ID":"d7a3de31-99d3-4527-ab2f-45db3660e884","Type":"ContainerStarted","Data":"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53"} Jul 16 20:41:25.245634 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:25.245610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wwdsb" event={"ID":"57422607-8d15-40bb-b7a9-807caa60e9a3","Type":"ContainerStarted","Data":"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2"} Jul 16 20:41:25.259783 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:25.259668 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-4d55p" podStartSLOduration=0.914676081 podStartE2EDuration="4.25964927s" totalImagesPullingTime="3.344973189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:25.259511279 +0000 UTC m=+483.760592010" watchObservedRunningTime="2026-07-16 20:41:25.25964927 +0000 UTC m=+483.760729997" Jul 16 20:41:25.279106 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:25.279053 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wwdsb" podStartSLOduration=0.764193546 podStartE2EDuration="4.279037616s" totalImagesPullingTime="3.51484407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:25.277173219 +0000 UTC m=+483.778253946" watchObservedRunningTime="2026-07-16 20:41:25.279037616 +0000 UTC m=+483.780118343" Jul 16 20:41:25.285270 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:25.285239 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:27.254371 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:27.254325 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wwdsb" podUID="57422607-8d15-40bb-b7a9-807caa60e9a3" containerName="authorino" containerID="cri-o://a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2" gracePeriod=30 Jul 16 20:41:27.509546 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:27.509478 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:27.558429 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:27.558395 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4hn6\" (UniqueName: \"kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6\") pod \"57422607-8d15-40bb-b7a9-807caa60e9a3\" (UID: \"57422607-8d15-40bb-b7a9-807caa60e9a3\") " Jul 16 20:41:27.560645 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:27.560610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6" (OuterVolumeSpecName: "kube-api-access-j4hn6") pod "57422607-8d15-40bb-b7a9-807caa60e9a3" (UID: "57422607-8d15-40bb-b7a9-807caa60e9a3"). InnerVolumeSpecName "kube-api-access-j4hn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:41:27.659389 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:27.659348 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hn6\" (UniqueName: \"kubernetes.io/projected/57422607-8d15-40bb-b7a9-807caa60e9a3-kube-api-access-j4hn6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:41:28.259974 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.259940 2564 generic.go:356] "Generic (PLEG): container finished" podID="57422607-8d15-40bb-b7a9-807caa60e9a3" containerID="a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2" exitCode=0 Jul 16 20:41:28.260395 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.259987 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wwdsb" Jul 16 20:41:28.260395 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.260030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wwdsb" event={"ID":"57422607-8d15-40bb-b7a9-807caa60e9a3","Type":"ContainerDied","Data":"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2"} Jul 16 20:41:28.260395 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.260066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wwdsb" event={"ID":"57422607-8d15-40bb-b7a9-807caa60e9a3","Type":"ContainerDied","Data":"daa5ebf334b5da0931b8d6f454aea32df1d07bfc5175b366ae8d2c0d384e0609"} Jul 16 20:41:28.260395 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.260083 2564 scope.go:117] "RemoveContainer" containerID="a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2" Jul 16 20:41:28.269003 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.268980 2564 scope.go:117] "RemoveContainer" containerID="a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2" Jul 16 20:41:28.269294 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:41:28.269274 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2\": container with ID starting with a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2 not found: ID does not exist" containerID="a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2" Jul 16 20:41:28.269388 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.269332 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2"} err="failed to get container status \"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2\": rpc error: code = NotFound desc = could not find container \"a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2\": container with ID starting with a1165f25e3816e631a2ff1f4435374842c48d3cd82849566dee575c797c455d2 not found: ID does not exist" Jul 16 20:41:28.279610 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.279582 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:28.283672 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:28.283635 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wwdsb"] Jul 16 20:41:30.039360 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:30.039322 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57422607-8d15-40bb-b7a9-807caa60e9a3" path="/var/lib/kubelet/pods/57422607-8d15-40bb-b7a9-807caa60e9a3/volumes" Jul 16 20:41:32.715852 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:32.715815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:32.765750 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:32.765707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:35.249413 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:35.249375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:35.351223 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:35.351180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:35.450956 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:35.450922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:36.348942 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:36.348896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:36.449428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:36.449382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:36.875712 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:36.875680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:39.121509 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:39.121477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:39.215890 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:39.215858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:39.722751 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:39.722712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:39.780112 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:39.780081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:40.978164 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:40.978102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:41.284134 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:41.284057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:43.116116 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:43.116077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:43.216031 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:43.215996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:49.029151 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.029114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:49.415752 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.415675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:49.513712 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.513679 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:49.514058 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.514044 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57422607-8d15-40bb-b7a9-807caa60e9a3" containerName="authorino" Jul 16 20:41:49.514058 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.514058 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57422607-8d15-40bb-b7a9-807caa60e9a3" containerName="authorino" Jul 16 20:41:49.514149 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.514130 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57422607-8d15-40bb-b7a9-807caa60e9a3" containerName="authorino" Jul 16 20:41:49.516218 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.516202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:49.522015 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.521989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:49.667410 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.667279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkwgh\" (UniqueName: \"kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh\") pod \"authorino-857fffc989-q5kkw\" (UID: \"9d16e8e2-11df-4da5-9abc-2683b691b913\") " pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:49.768489 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.768452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkwgh\" (UniqueName: \"kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh\") pod \"authorino-857fffc989-q5kkw\" (UID: \"9d16e8e2-11df-4da5-9abc-2683b691b913\") " pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:49.777916 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.777881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkwgh\" (UniqueName: \"kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh\") pod \"authorino-857fffc989-q5kkw\" (UID: \"9d16e8e2-11df-4da5-9abc-2683b691b913\") " pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:49.780814 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.780783 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:49.781092 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.781076 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:49.809669 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.809638 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:41:49.812660 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.812637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:49.815195 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.815167 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 20:41:49.819864 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.819837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:41:49.923937 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.923912 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:49.926681 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:41:49.926650 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d16e8e2_11df_4da5_9abc_2683b691b913.slice/crio-7208e70602f6a97f12b000fa9002e0485280a5a9ed284528b83177be675085a7 WatchSource:0}: Error finding container 7208e70602f6a97f12b000fa9002e0485280a5a9ed284528b83177be675085a7: Status 404 returned error can't find the container with id 7208e70602f6a97f12b000fa9002e0485280a5a9ed284528b83177be675085a7 Jul 16 20:41:49.970123 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.970076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:49.970296 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:49.970138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjswv\" (UniqueName: \"kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.071636 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.071594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.072097 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.071650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjswv\" (UniqueName: \"kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.074437 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.074412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.081887 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.081858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjswv\" (UniqueName: \"kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv\") pod \"authorino-669885c6b5-2tmq5\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.123765 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.123725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:41:50.262844 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.262818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:41:50.265096 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:41:50.265066 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9da5cbca_bd90_4da0_bb68_50234cc80072.slice/crio-6fb488e43f4fd6aa265290a6b54260c42fb1a5784020905b99cc328eb033a5c9 WatchSource:0}: Error finding container 6fb488e43f4fd6aa265290a6b54260c42fb1a5784020905b99cc328eb033a5c9: Status 404 returned error can't find the container with id 6fb488e43f4fd6aa265290a6b54260c42fb1a5784020905b99cc328eb033a5c9 Jul 16 20:41:50.368370 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.368329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669885c6b5-2tmq5" event={"ID":"9da5cbca-bd90-4da0-bb68-50234cc80072","Type":"ContainerStarted","Data":"6fb488e43f4fd6aa265290a6b54260c42fb1a5784020905b99cc328eb033a5c9"} Jul 16 20:41:50.369561 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:50.369535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857fffc989-q5kkw" event={"ID":"9d16e8e2-11df-4da5-9abc-2683b691b913","Type":"ContainerStarted","Data":"7208e70602f6a97f12b000fa9002e0485280a5a9ed284528b83177be675085a7"} Jul 16 20:41:51.379005 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.378959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857fffc989-q5kkw" event={"ID":"9d16e8e2-11df-4da5-9abc-2683b691b913","Type":"ContainerStarted","Data":"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50"} Jul 16 20:41:51.379492 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.378973 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-857fffc989-q5kkw" podUID="9d16e8e2-11df-4da5-9abc-2683b691b913" containerName="authorino" containerID="cri-o://e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50" gracePeriod=30 Jul 16 20:41:51.380608 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.380585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669885c6b5-2tmq5" event={"ID":"9da5cbca-bd90-4da0-bb68-50234cc80072","Type":"ContainerStarted","Data":"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120"} Jul 16 20:41:51.395408 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.395362 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-857fffc989-q5kkw" podStartSLOduration=1.8823111460000002 podStartE2EDuration="2.395344847s" totalImagesPullingTime="513.033701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:51.392934973 +0000 UTC m=+509.894015699" watchObservedRunningTime="2026-07-16 20:41:51.395344847 +0000 UTC m=+509.896425579" Jul 16 20:41:51.408259 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.408204 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-669885c6b5-2tmq5" podStartSLOduration=2.057480289 podStartE2EDuration="2.408186731s" totalImagesPullingTime="350.706442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:51.407916667 +0000 UTC m=+509.908997393" watchObservedRunningTime="2026-07-16 20:41:51.408186731 +0000 UTC m=+509.909267486" Jul 16 20:41:51.434382 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.434347 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:51.434583 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.434557 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-4d55p" podUID="d7a3de31-99d3-4527-ab2f-45db3660e884" containerName="authorino" containerID="cri-o://2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53" gracePeriod=30 Jul 16 20:41:51.683405 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.683376 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:51.709550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.709515 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:51.786959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.786924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkwgh\" (UniqueName: \"kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh\") pod \"9d16e8e2-11df-4da5-9abc-2683b691b913\" (UID: \"9d16e8e2-11df-4da5-9abc-2683b691b913\") " Jul 16 20:41:51.789146 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.789117 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh" (OuterVolumeSpecName: "kube-api-access-gkwgh") pod "9d16e8e2-11df-4da5-9abc-2683b691b913" (UID: "9d16e8e2-11df-4da5-9abc-2683b691b913"). InnerVolumeSpecName "kube-api-access-gkwgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:41:51.889170 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.888530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtvcn\" (UniqueName: \"kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn\") pod \"d7a3de31-99d3-4527-ab2f-45db3660e884\" (UID: \"d7a3de31-99d3-4527-ab2f-45db3660e884\") " Jul 16 20:41:51.889170 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.889067 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkwgh\" (UniqueName: \"kubernetes.io/projected/9d16e8e2-11df-4da5-9abc-2683b691b913-kube-api-access-gkwgh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:41:51.894977 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.894898 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn" (OuterVolumeSpecName: "kube-api-access-vtvcn") pod "d7a3de31-99d3-4527-ab2f-45db3660e884" (UID: "d7a3de31-99d3-4527-ab2f-45db3660e884"). InnerVolumeSpecName "kube-api-access-vtvcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:41:51.990592 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:51.990549 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvcn\" (UniqueName: \"kubernetes.io/projected/d7a3de31-99d3-4527-ab2f-45db3660e884-kube-api-access-vtvcn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:41:52.386549 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.386507 2564 generic.go:356] "Generic (PLEG): container finished" podID="d7a3de31-99d3-4527-ab2f-45db3660e884" containerID="2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53" exitCode=0 Jul 16 20:41:52.387037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.386567 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4d55p" Jul 16 20:41:52.387037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.386594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4d55p" event={"ID":"d7a3de31-99d3-4527-ab2f-45db3660e884","Type":"ContainerDied","Data":"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53"} Jul 16 20:41:52.387037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.386646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4d55p" event={"ID":"d7a3de31-99d3-4527-ab2f-45db3660e884","Type":"ContainerDied","Data":"dcc81b8e8cf56cc81a6049d5ce3bf9ac2db4b6a2f2d95877e971ac0a9932c69f"} Jul 16 20:41:52.387037 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.386670 2564 scope.go:117] "RemoveContainer" containerID="2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53" Jul 16 20:41:52.387902 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.387854 2564 generic.go:356] "Generic (PLEG): container finished" podID="9d16e8e2-11df-4da5-9abc-2683b691b913" containerID="e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50" exitCode=0 Jul 16 20:41:52.387973 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.387926 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857fffc989-q5kkw" Jul 16 20:41:52.387973 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.387936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857fffc989-q5kkw" event={"ID":"9d16e8e2-11df-4da5-9abc-2683b691b913","Type":"ContainerDied","Data":"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50"} Jul 16 20:41:52.388070 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.387971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857fffc989-q5kkw" event={"ID":"9d16e8e2-11df-4da5-9abc-2683b691b913","Type":"ContainerDied","Data":"7208e70602f6a97f12b000fa9002e0485280a5a9ed284528b83177be675085a7"} Jul 16 20:41:52.395463 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.395422 2564 scope.go:117] "RemoveContainer" containerID="2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53" Jul 16 20:41:52.395789 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:41:52.395755 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53\": container with ID starting with 2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53 not found: ID does not exist" containerID="2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53" Jul 16 20:41:52.395895 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.395800 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53"} err="failed to get container status \"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53\": rpc error: code = NotFound desc = could not find container \"2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53\": container with ID starting with 2a1a6447b7bfe6347bf9215154987f96ded8fe3450516a7f8d95fea9819b4a53 not found: ID does not exist" Jul 16 20:41:52.395895 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.395820 2564 scope.go:117] "RemoveContainer" containerID="e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50" Jul 16 20:41:52.403188 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.403160 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:52.403859 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.403842 2564 scope.go:117] "RemoveContainer" containerID="e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50" Jul 16 20:41:52.404151 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:41:52.404131 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50\": container with ID starting with e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50 not found: ID does not exist" containerID="e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50" Jul 16 20:41:52.404292 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.404160 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50"} err="failed to get container status \"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50\": rpc error: code = NotFound desc = could not find container \"e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50\": container with ID starting with e5456f4bb864a6a2956fada8bd9dfd9a0cc84884d3ec19d39e88d8dd04075f50 not found: ID does not exist" Jul 16 20:41:52.406377 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.406357 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4d55p"] Jul 16 20:41:52.416391 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.416362 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:52.419819 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:52.419796 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-857fffc989-q5kkw"] Jul 16 20:41:54.040076 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:54.040041 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d16e8e2-11df-4da5-9abc-2683b691b913" path="/var/lib/kubelet/pods/9d16e8e2-11df-4da5-9abc-2683b691b913/volumes" Jul 16 20:41:54.040484 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:54.040398 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a3de31-99d3-4527-ab2f-45db3660e884" path="/var/lib/kubelet/pods/d7a3de31-99d3-4527-ab2f-45db3660e884/volumes" Jul 16 20:41:56.132156 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:56.132122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:56.231818 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:56.231785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:56.306490 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:56.306456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:56.366597 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:56.366567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:59.425718 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:59.425684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:41:59.525435 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:41:59.525398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:11.261948 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:11.261913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:28.653453 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:28.653418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:28.750001 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:28.749969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:29.173320 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.173265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:29.327981 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.327945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:29.776653 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.776625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:29.885758 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.885721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:29.925035 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925001 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hjtkn"] Jul 16 20:42:29.925377 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925364 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a3de31-99d3-4527-ab2f-45db3660e884" containerName="authorino" Jul 16 20:42:29.925458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925378 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a3de31-99d3-4527-ab2f-45db3660e884" containerName="authorino" Jul 16 20:42:29.925458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925400 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d16e8e2-11df-4da5-9abc-2683b691b913" containerName="authorino" Jul 16 20:42:29.925458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925406 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d16e8e2-11df-4da5-9abc-2683b691b913" containerName="authorino" Jul 16 20:42:29.925562 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925466 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d16e8e2-11df-4da5-9abc-2683b691b913" containerName="authorino" Jul 16 20:42:29.925562 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.925477 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a3de31-99d3-4527-ab2f-45db3660e884" containerName="authorino" Jul 16 20:42:29.928420 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.928399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:29.931631 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.931606 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 20:42:29.931748 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.931691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8wh5b\"" Jul 16 20:42:29.954053 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:29.954029 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hjtkn"] Jul 16 20:42:30.031416 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.031347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3666a563-e014-4fbe-ad65-ef0a378cfa88-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.031547 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.031465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4c7\" (UniqueName: \"kubernetes.io/projected/3666a563-e014-4fbe-ad65-ef0a378cfa88-kube-api-access-wz4c7\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.132984 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.132946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3666a563-e014-4fbe-ad65-ef0a378cfa88-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.133157 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.133046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4c7\" (UniqueName: \"kubernetes.io/projected/3666a563-e014-4fbe-ad65-ef0a378cfa88-kube-api-access-wz4c7\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.133899 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.133877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3666a563-e014-4fbe-ad65-ef0a378cfa88-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.141215 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.141185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4c7\" (UniqueName: \"kubernetes.io/projected/3666a563-e014-4fbe-ad65-ef0a378cfa88-kube-api-access-wz4c7\") pod \"payload-processing-5bf47cd988-hjtkn\" (UID: \"3666a563-e014-4fbe-ad65-ef0a378cfa88\") " pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.239103 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.239068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:30.576948 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:30.576799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hjtkn"] Jul 16 20:42:30.579124 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:42:30.579098 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3666a563_e014_4fbe_ad65_ef0a378cfa88.slice/crio-11094f2b610c84957c5797f87cd082c52f537a3656016975f6a0ce7acdeb3df3 WatchSource:0}: Error finding container 11094f2b610c84957c5797f87cd082c52f537a3656016975f6a0ce7acdeb3df3: Status 404 returned error can't find the container with id 11094f2b610c84957c5797f87cd082c52f537a3656016975f6a0ce7acdeb3df3 Jul 16 20:42:31.581610 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:31.581576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" event={"ID":"3666a563-e014-4fbe-ad65-ef0a378cfa88","Type":"ContainerStarted","Data":"11094f2b610c84957c5797f87cd082c52f537a3656016975f6a0ce7acdeb3df3"} Jul 16 20:42:34.609228 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:34.609130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" event={"ID":"3666a563-e014-4fbe-ad65-ef0a378cfa88","Type":"ContainerStarted","Data":"fe4aa2eda332317f11a8fb83efa175228275b4d55ff43edc36e290c9c2a04931"} Jul 16 20:42:34.609727 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:34.609274 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:34.628293 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:34.628231 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" podStartSLOduration=1.889284408 podStartE2EDuration="5.628214985s" totalImagesPullingTime="3.738930577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:34.626263332 +0000 UTC m=+553.127344070" watchObservedRunningTime="2026-07-16 20:42:34.628214985 +0000 UTC m=+553.129295709" Jul 16 20:42:38.953125 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:38.953086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:39.051676 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.051644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:39.636024 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.635931 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a616930-461e-40da-8b27-e6052c45ccc0" containerID="21d0922ca901b36cf2a512937085b92c2546051a25b7ee3951c1ef45c0dc1aa1" exitCode=0 Jul 16 20:42:39.636024 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.636004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" event={"ID":"5a616930-461e-40da-8b27-e6052c45ccc0","Type":"ContainerDied","Data":"21d0922ca901b36cf2a512937085b92c2546051a25b7ee3951c1ef45c0dc1aa1"} Jul 16 20:42:39.642077 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.636958 2564 scope.go:117] "RemoveContainer" containerID="21d0922ca901b36cf2a512937085b92c2546051a25b7ee3951c1ef45c0dc1aa1" Jul 16 20:42:39.643011 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.642979 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:42:39.643087 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.643064 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:42:39.643134 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:39.643106 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:42:40.644606 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:40.644572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" event={"ID":"5a616930-461e-40da-8b27-e6052c45ccc0","Type":"ContainerStarted","Data":"7ae79016a33bb23ca8c7d9cba0de0e676291a6385ff15e382b05f0c15f8b7980"} Jul 16 20:42:40.789403 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:40.789357 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:42:40.789648 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:40.789620 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xfnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 16 20:42:40.789726 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:40.789696 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" podUID="5a616930-461e-40da-8b27-e6052c45ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 16 20:42:41.789608 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:41.789569 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xfnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 16 20:42:41.790179 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:41.790145 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" podUID="5a616930-461e-40da-8b27-e6052c45ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 16 20:42:42.789869 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:42.789829 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xfnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 16 20:42:42.790479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:42.790446 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" podUID="5a616930-461e-40da-8b27-e6052c45ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 16 20:42:43.789745 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:43.789708 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xfnzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 16 20:42:43.790027 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:43.789996 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" podUID="5a616930-461e-40da-8b27-e6052c45ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 16 20:42:44.792062 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:44.792031 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:42:44.792449 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:44.792292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:42:44.792911 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:44.792895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xfnzl" Jul 16 20:42:45.617414 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:45.617384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-hjtkn" Jul 16 20:42:52.620981 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:52.620953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:59.099397 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.099366 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc"] Jul 16 20:42:59.102039 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.102019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.104905 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.104884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 20:42:59.106002 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.105982 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 20:42:59.106353 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.106335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jqnfp\"" Jul 16 20:42:59.106522 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.106506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 20:42:59.113390 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.113370 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc"] Jul 16 20:42:59.186639 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.186602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:42:59.219915 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.219879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2nv\" (UniqueName: \"kubernetes.io/projected/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-kube-api-access-gq2nv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.220099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.219924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.220099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.219945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.220099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.220013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.220099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.220066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.220099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.220097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321253 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2nv\" (UniqueName: \"kubernetes.io/projected/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-kube-api-access-gq2nv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321500 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321536 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321575 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321641 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321777 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.321871 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.321850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.323814 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.323784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.323996 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.323980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.330621 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.330591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2nv\" (UniqueName: \"kubernetes.io/projected/5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461-kube-api-access-gq2nv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2kvwc\" (UID: \"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.411813 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.411727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:42:59.550312 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.550273 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc"] Jul 16 20:42:59.552436 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:42:59.552412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e09ce08_a4d0_48b1_8dd9_1d3ebb3b7461.slice/crio-bee50dd0bf3bc2fadb0ff63f7f16e8643f089a5f67bf031513d49f51d0b4bf6f WatchSource:0}: Error finding container bee50dd0bf3bc2fadb0ff63f7f16e8643f089a5f67bf031513d49f51d0b4bf6f: Status 404 returned error can't find the container with id bee50dd0bf3bc2fadb0ff63f7f16e8643f089a5f67bf031513d49f51d0b4bf6f Jul 16 20:42:59.739448 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:42:59.739412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" event={"ID":"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461","Type":"ContainerStarted","Data":"bee50dd0bf3bc2fadb0ff63f7f16e8643f089a5f67bf031513d49f51d0b4bf6f"} Jul 16 20:43:00.767149 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:00.767083 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" podStartSLOduration=0.664323706 podStartE2EDuration="1.767067879s" totalImagesPullingTime="1.102744173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:00.765352151 +0000 UTC m=+579.266432876" watchObservedRunningTime="2026-07-16 20:43:00.767067879 +0000 UTC m=+579.268148603" Jul 16 20:43:01.752906 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:01.752871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" event={"ID":"5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461","Type":"ContainerStarted","Data":"66e40053e772eb46c3f32feb72ed604db63ac71c30402a73f565a3c5feec2be1"} Jul 16 20:43:09.412065 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:09.412025 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:43:09.412666 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:09.412101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:43:09.424238 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:09.424211 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:43:09.801432 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:09.801401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2kvwc" Jul 16 20:43:11.085757 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.085727 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-988cr"] Jul 16 20:43:11.088616 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.088591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.091428 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.091403 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 20:43:11.100602 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.100579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-988cr"] Jul 16 20:43:11.119402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.119366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:43:11.233982 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.233949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmwr\" (UniqueName: \"kubernetes.io/projected/a06addce-875a-46e5-b2fb-2fd58f5c828b-kube-api-access-vbmwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.234154 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.234026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.234154 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.234044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06addce-875a-46e5-b2fb-2fd58f5c828b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.234154 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.234063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.234257 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.234160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.234257 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.234186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.334862 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmwr\" (UniqueName: \"kubernetes.io/projected/a06addce-875a-46e5-b2fb-2fd58f5c828b-kube-api-access-vbmwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06addce-875a-46e5-b2fb-2fd58f5c828b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335061 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.334995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335430 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.335409 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335487 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.335447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.335487 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.335470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.337460 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.337414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06addce-875a-46e5-b2fb-2fd58f5c828b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.337588 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.337573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06addce-875a-46e5-b2fb-2fd58f5c828b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.343577 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.343557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmwr\" (UniqueName: \"kubernetes.io/projected/a06addce-875a-46e5-b2fb-2fd58f5c828b-kube-api-access-vbmwr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-988cr\" (UID: \"a06addce-875a-46e5-b2fb-2fd58f5c828b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.398536 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.398508 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:11.528729 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.528615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-988cr"] Jul 16 20:43:11.531361 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:43:11.531322 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06addce_875a_46e5_b2fb_2fd58f5c828b.slice/crio-f7797b7b0acd2eaccb51cf74476aa4497087eb2513bf75b957a5778211dc24f8 WatchSource:0}: Error finding container f7797b7b0acd2eaccb51cf74476aa4497087eb2513bf75b957a5778211dc24f8: Status 404 returned error can't find the container with id f7797b7b0acd2eaccb51cf74476aa4497087eb2513bf75b957a5778211dc24f8 Jul 16 20:43:11.802367 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.802328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" event={"ID":"a06addce-875a-46e5-b2fb-2fd58f5c828b","Type":"ContainerStarted","Data":"6f1dd38634c2609c0f33f2ac121d997a445de4f206fb74db4fe50c5d5815132d"} Jul 16 20:43:11.802516 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.802388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" event={"ID":"a06addce-875a-46e5-b2fb-2fd58f5c828b","Type":"ContainerStarted","Data":"f7797b7b0acd2eaccb51cf74476aa4497087eb2513bf75b957a5778211dc24f8"} Jul 16 20:43:11.820976 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:11.820923 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" podStartSLOduration=0.622374991 podStartE2EDuration="820.907761ms" totalImagesPullingTime="198.53277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:11.819104837 +0000 UTC m=+590.320185563" watchObservedRunningTime="2026-07-16 20:43:11.820907761 +0000 UTC m=+590.321988484" Jul 16 20:43:16.682817 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:16.682780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:43:21.399389 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.399350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:21.399780 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.399417 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:21.411889 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.411860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:21.863508 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.863478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-988cr" Jul 16 20:43:21.913250 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.913226 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:43:21.913466 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:21.913231 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:43:22.082340 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:22.082294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:43:28.497999 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.497964 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs"] Jul 16 20:43:28.500658 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.500642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.504088 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.504070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 20:43:28.522193 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.522164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs"] Jul 16 20:43:28.559635 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.559605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:43:28.598969 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.598932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2l7\" (UniqueName: \"kubernetes.io/projected/c3dfe486-9ebb-4522-87f9-025c3de18ad9-kube-api-access-sm2l7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.599153 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.598995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.599153 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.599068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.599153 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.599099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.599153 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.599140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.599376 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.599164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.699991 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.699957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.699991 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.699998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700210 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700210 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700210 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700210 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2l7\" (UniqueName: \"kubernetes.io/projected/c3dfe486-9ebb-4522-87f9-025c3de18ad9-kube-api-access-sm2l7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700426 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700486 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.700595 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.700575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.702476 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.702457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3dfe486-9ebb-4522-87f9-025c3de18ad9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.702657 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.702640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3dfe486-9ebb-4522-87f9-025c3de18ad9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.710355 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.710327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2l7\" (UniqueName: \"kubernetes.io/projected/c3dfe486-9ebb-4522-87f9-025c3de18ad9-kube-api-access-sm2l7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs\" (UID: \"c3dfe486-9ebb-4522-87f9-025c3de18ad9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.809842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.809747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:28.947687 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:28.947665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs"] Jul 16 20:43:28.949790 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:43:28.949756 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3dfe486_9ebb_4522_87f9_025c3de18ad9.slice/crio-214b556446f5b3b89fee64bd16a7bf4534674194b87f37872acfc1ead80a26ba WatchSource:0}: Error finding container 214b556446f5b3b89fee64bd16a7bf4534674194b87f37872acfc1ead80a26ba: Status 404 returned error can't find the container with id 214b556446f5b3b89fee64bd16a7bf4534674194b87f37872acfc1ead80a26ba Jul 16 20:43:29.895947 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:29.895908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" event={"ID":"c3dfe486-9ebb-4522-87f9-025c3de18ad9","Type":"ContainerStarted","Data":"5f86085095d76cb0c4b3406d31fd4ac8e2f813d16c2f358321d533067922aa05"} Jul 16 20:43:29.896371 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:29.895957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" event={"ID":"c3dfe486-9ebb-4522-87f9-025c3de18ad9","Type":"ContainerStarted","Data":"214b556446f5b3b89fee64bd16a7bf4534674194b87f37872acfc1ead80a26ba"} Jul 16 20:43:29.919252 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:29.919199 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" podStartSLOduration=1.704109988 podStartE2EDuration="1.919183606s" totalImagesPullingTime="215.073618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:29.91744886 +0000 UTC m=+608.418529585" watchObservedRunningTime="2026-07-16 20:43:29.919183606 +0000 UTC m=+608.420264330" Jul 16 20:43:38.810026 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:38.809973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:38.810457 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:38.810044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:38.822402 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:38.822376 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:38.953463 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:38.953434 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs" Jul 16 20:43:47.968137 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:43:47.968006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:01.282546 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:01.282516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:19.568951 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.568912 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d66667464-lmjqj"] Jul 16 20:44:19.572849 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.572821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.583382 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.583354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d66667464-lmjqj"] Jul 16 20:44:19.689336 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.689263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f6b26-df01-410e-9f39-da359c8471ad-tls-cert\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.689522 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.689389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vxm2\" (UniqueName: \"kubernetes.io/projected/6e3f6b26-df01-410e-9f39-da359c8471ad-kube-api-access-7vxm2\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.789912 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.789871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vxm2\" (UniqueName: \"kubernetes.io/projected/6e3f6b26-df01-410e-9f39-da359c8471ad-kube-api-access-7vxm2\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.790105 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.789978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f6b26-df01-410e-9f39-da359c8471ad-tls-cert\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.792677 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.792651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f6b26-df01-410e-9f39-da359c8471ad-tls-cert\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.800190 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.800162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vxm2\" (UniqueName: \"kubernetes.io/projected/6e3f6b26-df01-410e-9f39-da359c8471ad-kube-api-access-7vxm2\") pod \"authorino-5d66667464-lmjqj\" (UID: \"6e3f6b26-df01-410e-9f39-da359c8471ad\") " pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.835739 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.835656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:19.883393 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.883354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d66667464-lmjqj" Jul 16 20:44:19.943085 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:19.943045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:20.052209 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:20.052185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d66667464-lmjqj"] Jul 16 20:44:20.054612 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:44:20.054584 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e3f6b26_df01_410e_9f39_da359c8471ad.slice/crio-6cd00df03f224bc12194db8392a7765c17e7819aeebc82a10f5c0e8a78b6c38c WatchSource:0}: Error finding container 6cd00df03f224bc12194db8392a7765c17e7819aeebc82a10f5c0e8a78b6c38c: Status 404 returned error can't find the container with id 6cd00df03f224bc12194db8392a7765c17e7819aeebc82a10f5c0e8a78b6c38c Jul 16 20:44:20.056254 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:20.056238 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 20:44:20.143462 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:20.143374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d66667464-lmjqj" event={"ID":"6e3f6b26-df01-410e-9f39-da359c8471ad","Type":"ContainerStarted","Data":"6cd00df03f224bc12194db8392a7765c17e7819aeebc82a10f5c0e8a78b6c38c"} Jul 16 20:44:21.092709 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.092651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:21.150603 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.150564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d66667464-lmjqj" event={"ID":"6e3f6b26-df01-410e-9f39-da359c8471ad","Type":"ContainerStarted","Data":"d05e0ae8d59fa78e073982eb2878c572eca98e5062edf8a0e8eff3fa94aa2497"} Jul 16 20:44:21.172540 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.172471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d66667464-lmjqj" podStartSLOduration=1.7042065210000001 podStartE2EDuration="2.172447813s" totalImagesPullingTime="468.241292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:44:21.172341085 +0000 UTC m=+659.673421811" watchObservedRunningTime="2026-07-16 20:44:21.172447813 +0000 UTC m=+659.673528535" Jul 16 20:44:21.222421 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.222387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:21.223084 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.223062 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:44:21.223275 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.223254 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-669885c6b5-2tmq5" podUID="9da5cbca-bd90-4da0-bb68-50234cc80072" containerName="authorino" containerID="cri-o://35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120" gracePeriod=30 Jul 16 20:44:21.489967 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.489931 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:44:21.621165 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.612658 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert\") pod \"9da5cbca-bd90-4da0-bb68-50234cc80072\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " Jul 16 20:44:21.621165 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.612711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjswv\" (UniqueName: \"kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv\") pod \"9da5cbca-bd90-4da0-bb68-50234cc80072\" (UID: \"9da5cbca-bd90-4da0-bb68-50234cc80072\") " Jul 16 20:44:21.621165 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.620453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv" (OuterVolumeSpecName: "kube-api-access-vjswv") pod "9da5cbca-bd90-4da0-bb68-50234cc80072" (UID: "9da5cbca-bd90-4da0-bb68-50234cc80072"). InnerVolumeSpecName "kube-api-access-vjswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:44:21.635850 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.635812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9da5cbca-bd90-4da0-bb68-50234cc80072" (UID: "9da5cbca-bd90-4da0-bb68-50234cc80072"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 20:44:21.714267 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.714178 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9da5cbca-bd90-4da0-bb68-50234cc80072-tls-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:44:21.714267 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.714211 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjswv\" (UniqueName: \"kubernetes.io/projected/9da5cbca-bd90-4da0-bb68-50234cc80072-kube-api-access-vjswv\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:44:21.849649 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:21.849619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:22.160517 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.160476 2564 generic.go:356] "Generic (PLEG): container finished" podID="9da5cbca-bd90-4da0-bb68-50234cc80072" containerID="35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120" exitCode=0 Jul 16 20:44:22.161144 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.161121 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669885c6b5-2tmq5" Jul 16 20:44:22.161667 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.161641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669885c6b5-2tmq5" event={"ID":"9da5cbca-bd90-4da0-bb68-50234cc80072","Type":"ContainerDied","Data":"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120"} Jul 16 20:44:22.161789 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.161685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669885c6b5-2tmq5" event={"ID":"9da5cbca-bd90-4da0-bb68-50234cc80072","Type":"ContainerDied","Data":"6fb488e43f4fd6aa265290a6b54260c42fb1a5784020905b99cc328eb033a5c9"} Jul 16 20:44:22.161789 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.161707 2564 scope.go:117] "RemoveContainer" containerID="35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120" Jul 16 20:44:22.173146 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.173125 2564 scope.go:117] "RemoveContainer" containerID="35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120" Jul 16 20:44:22.173546 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:44:22.173522 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120\": container with ID starting with 35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120 not found: ID does not exist" containerID="35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120" Jul 16 20:44:22.173644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.173555 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120"} err="failed to get container status \"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120\": rpc error: code = NotFound desc = could not find container \"35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120\": container with ID starting with 35107c5010d05583d0c6c057322b55c3211ac04768212b9ac1462006c6251120 not found: ID does not exist" Jul 16 20:44:22.195342 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.195264 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:44:22.200368 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:22.200331 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-669885c6b5-2tmq5"] Jul 16 20:44:24.039393 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:24.039358 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da5cbca-bd90-4da0-bb68-50234cc80072" path="/var/lib/kubelet/pods/9da5cbca-bd90-4da0-bb68-50234cc80072/volumes" Jul 16 20:44:40.605550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:40.605514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:44:40.762198 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:44:40.762158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:45:45.883161 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:45:45.883125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:46:25.442123 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:46:25.442077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:46:26.677749 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:46:26.677706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:02.268204 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:02.268102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:02.402859 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:02.402806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:02.523711 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:02.523619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:08.418547 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:08.418513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:08.523530 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:08.523493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:08.619404 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:08.619373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:08.789184 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:08.789154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:14.542250 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:14.542212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:47:14.604142 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:47:14.604096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:48:21.950272 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:48:21.950199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:48:21.950272 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:48:21.950199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:48:38.459405 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:48:38.459359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:48:39.654167 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:48:39.654119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:49:42.355267 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:49:42.355194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:51:17.587552 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:17.587427 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:51:17.588159 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:17.587709 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" podUID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" containerName="manager" containerID="cri-o://d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e" gracePeriod=10 Jul 16 20:51:21.146479 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.146448 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:51:21.199587 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.199556 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume\") pod \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " Jul 16 20:51:21.199771 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.199708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2dhb\" (UniqueName: \"kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb\") pod \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\" (UID: \"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd\") " Jul 16 20:51:21.199978 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.199952 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" (UID: "be4e08b6-f3a7-40bd-9564-3c3ff0d246dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 20:51:21.201939 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.201915 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb" (OuterVolumeSpecName: "kube-api-access-v2dhb") pod "be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" (UID: "be4e08b6-f3a7-40bd-9564-3c3ff0d246dd"). InnerVolumeSpecName "kube-api-access-v2dhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 20:51:21.286643 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.286549 2564 generic.go:356] "Generic (PLEG): container finished" podID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" containerID="d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e" exitCode=0 Jul 16 20:51:21.286643 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.286590 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" Jul 16 20:51:21.286643 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.286629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" event={"ID":"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd","Type":"ContainerDied","Data":"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e"} Jul 16 20:51:21.286989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.286687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4" event={"ID":"be4e08b6-f3a7-40bd-9564-3c3ff0d246dd","Type":"ContainerDied","Data":"72b7817cedcaf02a8848f98cf4b1cd8c9f6744926858b7c1d373d11ed72f60db"} Jul 16 20:51:21.286989 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.286719 2564 scope.go:117] "RemoveContainer" containerID="d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e" Jul 16 20:51:21.296998 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.296978 2564 scope.go:117] "RemoveContainer" containerID="d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e" Jul 16 20:51:21.297250 ip-10-0-131-179 kubenswrapper[2564]: E0716 20:51:21.297233 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e\": container with ID starting with d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e not found: ID does not exist" containerID="d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e" Jul 16 20:51:21.297291 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.297259 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e"} err="failed to get container status \"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e\": rpc error: code = NotFound desc = could not find container \"d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e\": container with ID starting with d0472a3ba2e22c77251688acb2dbfabdae277f32bf843892653fbc19d813390e not found: ID does not exist" Jul 16 20:51:21.300381 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.300363 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-extensions-socket-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:51:21.300438 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.300384 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dhb\" (UniqueName: \"kubernetes.io/projected/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd-kube-api-access-v2dhb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 20:51:21.318644 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.318614 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:51:21.320450 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:21.320427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8pnr4"] Jul 16 20:51:22.039537 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:51:22.039508 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" path="/var/lib/kubelet/pods/be4e08b6-f3a7-40bd-9564-3c3ff0d246dd/volumes" Jul 16 20:52:18.720477 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720442 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2"] Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720815 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da5cbca-bd90-4da0-bb68-50234cc80072" containerName="authorino" Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720826 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da5cbca-bd90-4da0-bb68-50234cc80072" containerName="authorino" Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720837 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" containerName="manager" Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720843 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" containerName="manager" Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720910 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4e08b6-f3a7-40bd-9564-3c3ff0d246dd" containerName="manager" Jul 16 20:52:18.720940 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.720918 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da5cbca-bd90-4da0-bb68-50234cc80072" containerName="authorino" Jul 16 20:52:18.724017 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.724000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.726751 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.726729 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mb4tc\"" Jul 16 20:52:18.738847 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.738824 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2"] Jul 16 20:52:18.829661 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.829628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtgk\" (UniqueName: \"kubernetes.io/projected/48aeacaa-0e70-4a85-91ba-7a59aebc381b-kube-api-access-kmtgk\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.829846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.829771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aeacaa-0e70-4a85-91ba-7a59aebc381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.930487 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.930442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aeacaa-0e70-4a85-91ba-7a59aebc381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.930666 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.930514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtgk\" (UniqueName: \"kubernetes.io/projected/48aeacaa-0e70-4a85-91ba-7a59aebc381b-kube-api-access-kmtgk\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.930809 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.930786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48aeacaa-0e70-4a85-91ba-7a59aebc381b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:18.940442 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:18.940412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtgk\" (UniqueName: \"kubernetes.io/projected/48aeacaa-0e70-4a85-91ba-7a59aebc381b-kube-api-access-kmtgk\") pod \"kuadrant-operator-controller-manager-86878c769f-qrdc2\" (UID: \"48aeacaa-0e70-4a85-91ba-7a59aebc381b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:19.033941 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.033849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:19.177837 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.177812 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2"] Jul 16 20:52:19.180136 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:52:19.180111 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aeacaa_0e70_4a85_91ba_7a59aebc381b.slice/crio-4c909910044cbef731f2547ee9a1e6da8e5b027d4bb56c3438c41f3d4492888d WatchSource:0}: Error finding container 4c909910044cbef731f2547ee9a1e6da8e5b027d4bb56c3438c41f3d4492888d: Status 404 returned error can't find the container with id 4c909910044cbef731f2547ee9a1e6da8e5b027d4bb56c3438c41f3d4492888d Jul 16 20:52:19.564099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.564057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" event={"ID":"48aeacaa-0e70-4a85-91ba-7a59aebc381b","Type":"ContainerStarted","Data":"ee18f4581a2f3af2d5ee52f4fc7a3ee88600140b7a709c285163f12b0f09b2a1"} Jul 16 20:52:19.564277 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.564111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" event={"ID":"48aeacaa-0e70-4a85-91ba-7a59aebc381b","Type":"ContainerStarted","Data":"4c909910044cbef731f2547ee9a1e6da8e5b027d4bb56c3438c41f3d4492888d"} Jul 16 20:52:19.564277 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.564145 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:19.592458 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:19.592401 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" podStartSLOduration=1.5923824500000001 podStartE2EDuration="1.59238245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:52:19.590771596 +0000 UTC m=+1138.091852324" watchObservedRunningTime="2026-07-16 20:52:19.59238245 +0000 UTC m=+1138.093463176" Jul 16 20:52:20.296710 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:20.296674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:52:20.375775 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:20.375743 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:52:30.570852 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:30.570812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrdc2" Jul 16 20:52:49.756104 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:52:49.756012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:53:21.986261 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:53:21.986228 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:53:21.988678 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:53:21.986764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:53:22.274788 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:53:22.274677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:54:27.779682 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:54:27.779647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:54:29.356104 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:54:29.356062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:57:19.085603 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:57:19.085510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:14.341350 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:14.341296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:22.016653 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:22.016518 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:58:22.020842 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:22.020105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 20:58:22.980516 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:22.980484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:31.260975 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:31.260941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:36.423287 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:36.423247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:39.674824 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:39.674784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:40.008272 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:40.008238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:40.184352 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:40.184313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:45.942288 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:45.942176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:46.094400 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.094356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:46.134600 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.134558 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 20:58:46.138113 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.138096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.141765 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.141742 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-866d7766-openshift-default-dockercfg-ptntk\"" Jul 16 20:58:46.163915 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.163883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 20:58:46.181387 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.181357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:46.218729 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm9c\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.218897 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.218897 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.218897 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.219035 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.219035 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.219035 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.218982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.219188 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.219071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.219188 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.219108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm9c\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320550 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320866 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320866 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320866 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.320866 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.321099 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.320884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.321405 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.321383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.323344 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.323292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.323344 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.323332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.329472 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.329443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.329584 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.329542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm9c\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c\") pod \"e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.448236 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.448198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:46.595152 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.595118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 20:58:46.597653 ip-10-0-131-179 kubenswrapper[2564]: W0716 20:58:46.597620 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6048a6_9ba9_407a_9d67_7be39e579337.slice/crio-b6e4ec2461c9eac4de52246ef034f0e85171c87918e8726a6315ce92b0bf2cc0 WatchSource:0}: Error finding container b6e4ec2461c9eac4de52246ef034f0e85171c87918e8726a6315ce92b0bf2cc0: Status 404 returned error can't find the container with id b6e4ec2461c9eac4de52246ef034f0e85171c87918e8726a6315ce92b0bf2cc0 Jul 16 20:58:46.599746 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.599714 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:58:46.599851 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.599778 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:58:46.599851 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:46.599807 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:58:47.407560 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.407524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:47.511260 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.511227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:47.569342 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.569292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerStarted","Data":"62b4de3ead5ecef06b2b464b106d11579bd537d4c72ebdf3838a05ee19dbe4ae"} Jul 16 20:58:47.569606 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.569584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerStarted","Data":"b6e4ec2461c9eac4de52246ef034f0e85171c87918e8726a6315ce92b0bf2cc0"} Jul 16 20:58:47.597672 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.597636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:47.598485 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.598444 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podStartSLOduration=1.5984268369999999 podStartE2EDuration="1.598426837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:58:47.59688051 +0000 UTC m=+1526.097961238" watchObservedRunningTime="2026-07-16 20:58:47.598426837 +0000 UTC m=+1526.099507562" Jul 16 20:58:47.660248 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:47.660162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:48.091068 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.091036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:48.275316 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.275259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:48.449237 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.449135 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:48.454175 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.454150 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:48.574238 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.574204 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:48.575132 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:48.575112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:58:49.873270 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:49.873200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:58:59.637316 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.637264 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerID="62b4de3ead5ecef06b2b464b106d11579bd537d4c72ebdf3838a05ee19dbe4ae" exitCode=0 Jul 16 20:58:59.637749 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.637341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerDied","Data":"62b4de3ead5ecef06b2b464b106d11579bd537d4c72ebdf3838a05ee19dbe4ae"} Jul 16 20:58:59.637794 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.637755 2564 scope.go:117] "RemoveContainer" containerID="62b4de3ead5ecef06b2b464b106d11579bd537d4c72ebdf3838a05ee19dbe4ae" Jul 16 20:58:59.638870 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.638843 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:58:59.638946 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.638929 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:58:59.638990 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:58:59.638977 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 20:59:00.644719 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:00.644680 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerStarted","Data":"0706640db65968f9fd0640540888b184c011a7de73b5e69c99215dd28308353f"} Jul 16 20:59:01.449207 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:01.449159 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:59:01.450279 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:01.450244 2564 patch_prober.go:28] interesting pod/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 20:59:01.450446 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:01.450312 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 20:59:02.449442 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:02.449403 2564 patch_prober.go:28] interesting pod/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 20:59:02.449846 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:02.449509 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 20:59:03.448778 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:03.448740 2564 patch_prober.go:28] interesting pod/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 20:59:03.448959 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:03.448804 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 20:59:03.574738 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:03.574680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:59:04.448820 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:04.448781 2564 patch_prober.go:28] interesting pod/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 20:59:04.449020 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:04.448842 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 20:59:05.453015 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:05.452984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:59:05.454067 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:05.454048 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 20:59:28.101233 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:28.101199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:59:52.540612 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:52.540579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:59:57.622800 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:57.622750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:59:57.765880 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:57.765846 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 20:59:57.766110 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:57.766088 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" containerID="cri-o://0706640db65968f9fd0640540888b184c011a7de73b5e69c99215dd28308353f" gracePeriod=30 Jul 16 20:59:57.788514 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:57.788482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 20:59:57.900803 ip-10-0-131-179 kubenswrapper[2564]: I0716 20:59:57.900709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:02.971332 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:02.971277 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerID="0706640db65968f9fd0640540888b184c011a7de73b5e69c99215dd28308353f" exitCode=0 Jul 16 21:00:02.971743 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:02.971333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerDied","Data":"0706640db65968f9fd0640540888b184c011a7de73b5e69c99215dd28308353f"} Jul 16 21:00:02.971743 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:02.971400 2564 scope.go:117] "RemoveContainer" containerID="62b4de3ead5ecef06b2b464b106d11579bd537d4c72ebdf3838a05ee19dbe4ae" Jul 16 21:00:03.029319 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.029235 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 21:00:03.085358 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkm9c\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085490 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085369 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085490 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085405 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085490 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085661 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085661 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085661 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085661 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085844 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo\") pod \"1f6048a6-9ba9-407a-9d67-7be39e579337\" (UID: \"1f6048a6-9ba9-407a-9d67-7be39e579337\") " Jul 16 21:00:03.085844 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085744 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:00:03.085844 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085755 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:00:03.085980 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.085927 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:00:03.086058 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.086015 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.086058 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.086036 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.086164 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.086068 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.086638 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.086572 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:00:03.086923 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.086656 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:00:03.088401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.088370 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:00:03.088552 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.088467 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c" (OuterVolumeSpecName: "kube-api-access-fkm9c") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "kube-api-access-fkm9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:00:03.088552 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.088492 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:00:03.089130 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.089099 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f6048a6-9ba9-407a-9d67-7be39e579337" (UID: "1f6048a6-9ba9-407a-9d67-7be39e579337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:00:03.187267 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187232 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6048a6-9ba9-407a-9d67-7be39e579337-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.187267 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187265 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.187267 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187274 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.187521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187281 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.187521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187290 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkm9c\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-kube-api-access-fkm9c\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.187521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.187314 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6048a6-9ba9-407a-9d67-7be39e579337-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:00:03.957025 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.956997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:03.978121 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.978087 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" Jul 16 21:00:03.978616 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.978126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx" event={"ID":"1f6048a6-9ba9-407a-9d67-7be39e579337","Type":"ContainerDied","Data":"b6e4ec2461c9eac4de52246ef034f0e85171c87918e8726a6315ce92b0bf2cc0"} Jul 16 21:00:03.978616 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:03.978209 2564 scope.go:117] "RemoveContainer" containerID="0706640db65968f9fd0640540888b184c011a7de73b5e69c99215dd28308353f" Jul 16 21:00:04.010022 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.009977 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 21:00:04.017632 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.017597 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-866d7766-openshift-default-f8fbb84c-jfxvx"] Jul 16 21:00:04.040397 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.040367 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" path="/var/lib/kubelet/pods/1f6048a6-9ba9-407a-9d67-7be39e579337/volumes" Jul 16 21:00:04.060724 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.060690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:04.069126 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069092 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:00:04.069582 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069568 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.069629 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069585 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.069629 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069598 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.069629 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069603 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.069729 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069665 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.069729 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.069674 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6048a6-9ba9-407a-9d67-7be39e579337" containerName="istio-proxy" Jul 16 21:00:04.074282 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.074253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.077174 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.077152 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-18adf534-gw-openshift-default-dockercfg-f54kn\"" Jul 16 21:00:04.085744 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.085715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:00:04.132984 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.132944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:04.197127 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197334 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197334 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197459 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197459 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197459 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197594 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnl6j\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197594 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.197594 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.197543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnl6j\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298967 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298967 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298967 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.298967 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.298967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299188 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299188 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299188 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299385 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299385 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299497 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299497 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.299696 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.299673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.301483 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.301461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.301719 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.301696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.307491 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.307462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.307656 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.307605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnl6j\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j\") pod \"e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.388068 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.388030 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:04.433798 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.433765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:04.541027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.540988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:04.562928 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.562902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:00:04.565278 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:00:04.565251 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf2e5c1_e817_4f73_9bdf_626cacd12486.slice/crio-14226b3d22cdb9bfe661eaa99917cbe0fd7c429ea360b9fff0bda5cdf3e63dd2 WatchSource:0}: Error finding container 14226b3d22cdb9bfe661eaa99917cbe0fd7c429ea360b9fff0bda5cdf3e63dd2: Status 404 returned error can't find the container with id 14226b3d22cdb9bfe661eaa99917cbe0fd7c429ea360b9fff0bda5cdf3e63dd2 Jul 16 21:00:04.567777 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.567735 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:04.567905 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.567822 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:04.567905 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.567863 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:04.986373 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.986335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerStarted","Data":"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343"} Jul 16 21:00:04.986847 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:04.986396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerStarted","Data":"14226b3d22cdb9bfe661eaa99917cbe0fd7c429ea360b9fff0bda5cdf3e63dd2"} Jul 16 21:00:05.009465 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.009408 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podStartSLOduration=1.009389591 podStartE2EDuration="1.009389591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:00:05.007422796 +0000 UTC m=+1603.508503520" watchObservedRunningTime="2026-07-16 21:00:05.009389591 +0000 UTC m=+1603.510470317" Jul 16 21:00:05.310660 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.310569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:05.388251 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.388215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:05.389666 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.389636 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:05.389831 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.389699 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:05.415579 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.415547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:05.603220 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.603105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:05.967007 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:05.966923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:06.165758 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:06.165725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:06.392688 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:06.392657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:06.997563 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:06.997530 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:06.998627 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:06.998608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:07.940656 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:07.940625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:21.071253 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.071168 2564 generic.go:356] "Generic (PLEG): container finished" podID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerID="6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343" exitCode=0 Jul 16 21:00:21.071253 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.071241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerDied","Data":"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343"} Jul 16 21:00:21.071717 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.071684 2564 scope.go:117] "RemoveContainer" containerID="6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343" Jul 16 21:00:21.072890 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.072853 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:21.072968 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.072946 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:21.073010 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.072993 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:00:21.998444 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:21.998398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:22.078676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:22.078630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerStarted","Data":"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6"} Jul 16 21:00:22.389006 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:22.388901 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:22.390201 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:22.390171 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:22.390338 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:22.390238 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:23.389068 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:23.389013 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:23.389754 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:23.389721 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:24.389427 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:24.389382 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:24.389932 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:24.389466 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:25.063189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:25.063161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:00:25.389260 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:25.389148 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:25.389260 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:25.389238 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:26.388791 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:26.388749 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 21:00:26.389028 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:26.388828 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 21:00:27.393212 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:27.393179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:27.393692 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:27.393547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:27.394431 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:27.394412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:00:46.072003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:00:46.071967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:09.959296 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:09.959257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:09.975975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:09.975937 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:01:09.976245 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:09.976207 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" containerID="cri-o://c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6" gracePeriod=30 Jul 16 21:01:11.419596 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:11.419568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:11.518810 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:11.518778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:12.396681 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:12.396642 2564 patch_prober.go:28] interesting pod/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:01:12.396881 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:12.396721 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:01:15.244022 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.243960 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:01:15.309422 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309613 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309613 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309501 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309613 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309521 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309613 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309572 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309839 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309839 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309642 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309839 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.309839 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnl6j\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j\") pod \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\" (UID: \"bcf2e5c1-e817-4f73-9bdf-626cacd12486\") " Jul 16 21:01:15.310235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.309999 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:01:15.310235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:01:15.310235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310115 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.310235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310136 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.310484 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:01:15.310484 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310459 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:01:15.310956 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.310770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:01:15.312324 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.312278 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:01:15.312604 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.312582 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:01:15.312666 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.312599 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j" (OuterVolumeSpecName: "kube-api-access-bnl6j") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "kube-api-access-bnl6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:01:15.312757 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.312741 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcf2e5c1-e817-4f73-9bdf-626cacd12486" (UID: "bcf2e5c1-e817-4f73-9bdf-626cacd12486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:01:15.356958 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.356921 2564 generic.go:356] "Generic (PLEG): container finished" podID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerID="c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6" exitCode=0 Jul 16 21:01:15.357141 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.356974 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" Jul 16 21:01:15.357141 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.357015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerDied","Data":"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6"} Jul 16 21:01:15.357141 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.357109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m" event={"ID":"bcf2e5c1-e817-4f73-9bdf-626cacd12486","Type":"ContainerDied","Data":"14226b3d22cdb9bfe661eaa99917cbe0fd7c429ea360b9fff0bda5cdf3e63dd2"} Jul 16 21:01:15.357141 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.357138 2564 scope.go:117] "RemoveContainer" containerID="c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6" Jul 16 21:01:15.366659 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.366642 2564 scope.go:117] "RemoveContainer" containerID="6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343" Jul 16 21:01:15.375331 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.375281 2564 scope.go:117] "RemoveContainer" containerID="c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6" Jul 16 21:01:15.375668 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:01:15.375646 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6\": container with ID starting with c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6 not found: ID does not exist" containerID="c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6" Jul 16 21:01:15.375826 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.375678 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6"} err="failed to get container status \"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6\": rpc error: code = NotFound desc = could not find container \"c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6\": container with ID starting with c233674c9e102c3976ab06f7fdfae7dbb4a8f8efeab6b72f0eec58b50f8b4aa6 not found: ID does not exist" Jul 16 21:01:15.375826 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.375696 2564 scope.go:117] "RemoveContainer" containerID="6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343" Jul 16 21:01:15.376048 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:01:15.376019 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343\": container with ID starting with 6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343 not found: ID does not exist" containerID="6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343" Jul 16 21:01:15.376123 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.376048 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343"} err="failed to get container status \"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343\": rpc error: code = NotFound desc = could not find container \"6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343\": container with ID starting with 6359817db59a7553eb0ff7b483a3762e9e1407554a6c081a74aca12779eaa343 not found: ID does not exist" Jul 16 21:01:15.378796 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.378775 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:01:15.382501 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.382477 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-18adf534-gw-openshift-default-654cd6b759-w9f8m"] Jul 16 21:01:15.410933 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410905 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.410933 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410933 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.411165 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410948 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.411165 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410961 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.411165 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410971 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.411165 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410979 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf2e5c1-e817-4f73-9bdf-626cacd12486-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.411165 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.410988 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnl6j\" (UniqueName: \"kubernetes.io/projected/bcf2e5c1-e817-4f73-9bdf-626cacd12486-kube-api-access-bnl6j\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:01:15.841560 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.841522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:15.944176 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:15.944143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:16.040793 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.040759 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" path="/var/lib/kubelet/pods/bcf2e5c1-e817-4f73-9bdf-626cacd12486/volumes" Jul 16 21:01:16.041086 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041067 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:01:16.041558 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041541 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.041638 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041559 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.041638 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041569 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.041638 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041575 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.041794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041644 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.041794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.041651 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf2e5c1-e817-4f73-9bdf-626cacd12486" containerName="istio-proxy" Jul 16 21:01:16.044916 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.044898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.047932 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.047710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-25df27a3-openshift-default-dockercfg-qkb9c\"" Jul 16 21:01:16.054241 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.054031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:01:16.075171 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.075138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:16.117010 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.116920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117010 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptzl\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.117532 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.117354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptzl\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.218840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.219053 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.218932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.219053 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.219024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.219129 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.219098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.221047 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.221019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.221245 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.221227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.226641 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.226618 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.226837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.226819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptzl\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl\") pod \"e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.358226 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.358146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:16.396850 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.396811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:16.483640 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.483604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:16.519101 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.519072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:01:16.520686 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:01:16.520658 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5398fdd_ce1a_428b_ab6b_e902f65c6857.slice/crio-e1c422f6fe3e02e10c01c73841d45827c7002892b8a80c180147a780d31f1ca1 WatchSource:0}: Error finding container e1c422f6fe3e02e10c01c73841d45827c7002892b8a80c180147a780d31f1ca1: Status 404 returned error can't find the container with id e1c422f6fe3e02e10c01c73841d45827c7002892b8a80c180147a780d31f1ca1 Jul 16 21:01:16.522886 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.522860 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:16.522975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.522926 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:16.523036 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:16.522976 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:17.302197 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.302157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:17.372182 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.372143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerStarted","Data":"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2"} Jul 16 21:01:17.372660 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.372209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerStarted","Data":"e1c422f6fe3e02e10c01c73841d45827c7002892b8a80c180147a780d31f1ca1"} Jul 16 21:01:17.428563 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.428504 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podStartSLOduration=1.428480905 podStartE2EDuration="1.428480905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:01:17.425326954 +0000 UTC m=+1675.926407672" watchObservedRunningTime="2026-07-16 21:01:17.428480905 +0000 UTC m=+1675.929561632" Jul 16 21:01:17.484869 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.484823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:17.599081 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:17.598987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:18.082914 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.082876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:18.190892 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.190851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:18.358818 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.358718 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:18.363762 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.363733 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:18.377594 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.377556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:18.378664 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:18.378643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:19.649957 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:19.649925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:29.444451 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.444356 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerID="84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2" exitCode=0 Jul 16 21:01:29.444451 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.444433 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerDied","Data":"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2"} Jul 16 21:01:29.445353 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.445289 2564 scope.go:117] "RemoveContainer" containerID="84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2" Jul 16 21:01:29.447716 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.447660 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:29.447863 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.447834 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:29.447976 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:29.447951 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:01:30.451507 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:30.451474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerStarted","Data":"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb"} Jul 16 21:01:31.359247 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:31.359203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:31.360442 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:31.360419 2564 patch_prober.go:28] interesting pod/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 21:01:31.360503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:31.360468 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 21:01:32.358906 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:32.358868 2564 patch_prober.go:28] interesting pod/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 21:01:32.359401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:32.358934 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 21:01:33.358863 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:33.358822 2564 patch_prober.go:28] interesting pod/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 21:01:33.359082 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:33.358906 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 21:01:33.377769 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:33.377720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:34.358851 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:34.358819 2564 patch_prober.go:28] interesting pod/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 21:01:34.359032 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:34.358878 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 21:01:35.363622 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:35.363588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:35.364573 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:35.364553 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:01:49.480388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:49.480290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:01:49.579533 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:01:49.579495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:32.686906 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:32.686863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:37.807292 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:37.807255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:39.013170 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:39.013132 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:02:39.013651 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:39.013379 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" containerID="cri-o://d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb" gracePeriod=30 Jul 16 21:02:39.044202 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:39.039350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:39.136927 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:39.136877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:44.286008 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.285931 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:02:44.393409 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393375 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ptzl\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393430 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393505 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393568 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393911 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393911 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393668 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393911 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393703 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert\") pod \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\" (UID: \"f5398fdd-ce1a-428b-ab6b-e902f65c6857\") " Jul 16 21:02:44.393911 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:02:44.393911 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.393822 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:02:44.394187 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.394148 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:02:44.394247 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.394180 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.394316 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.394214 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.394478 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.394432 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:02:44.394609 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.394584 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:02:44.396410 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.396369 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:02:44.396537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.396470 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl" (OuterVolumeSpecName: "kube-api-access-4ptzl") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "kube-api-access-4ptzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:02:44.396537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.396513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:02:44.396706 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.396690 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5398fdd-ce1a-428b-ab6b-e902f65c6857" (UID: "f5398fdd-ce1a-428b-ab6b-e902f65c6857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:02:44.495277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495236 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495270 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495280 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495288 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495592 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495297 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495592 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495341 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ptzl\" (UniqueName: \"kubernetes.io/projected/f5398fdd-ce1a-428b-ab6b-e902f65c6857-kube-api-access-4ptzl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.495592 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.495351 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5398fdd-ce1a-428b-ab6b-e902f65c6857-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:02:44.847082 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.847050 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerID="d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb" exitCode=0 Jul 16 21:02:44.847274 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.847102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerDied","Data":"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb"} Jul 16 21:02:44.847274 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.847147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" event={"ID":"f5398fdd-ce1a-428b-ab6b-e902f65c6857","Type":"ContainerDied","Data":"e1c422f6fe3e02e10c01c73841d45827c7002892b8a80c180147a780d31f1ca1"} Jul 16 21:02:44.847274 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.847169 2564 scope.go:117] "RemoveContainer" containerID="d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb" Jul 16 21:02:44.847274 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.847103 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz" Jul 16 21:02:44.857731 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.857708 2564 scope.go:117] "RemoveContainer" containerID="84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2" Jul 16 21:02:44.866931 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.866907 2564 scope.go:117] "RemoveContainer" containerID="d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb" Jul 16 21:02:44.867366 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:02:44.867335 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb\": container with ID starting with d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb not found: ID does not exist" containerID="d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb" Jul 16 21:02:44.867494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.867374 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb"} err="failed to get container status \"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb\": rpc error: code = NotFound desc = could not find container \"d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb\": container with ID starting with d88cbe91399fb4bf3b343d49524639a5bcdd4af4053a0e1dc64ee0fc0c4c95cb not found: ID does not exist" Jul 16 21:02:44.867494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.867399 2564 scope.go:117] "RemoveContainer" containerID="84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2" Jul 16 21:02:44.867975 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:02:44.867940 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2\": container with ID starting with 84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2 not found: ID does not exist" containerID="84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2" Jul 16 21:02:44.868077 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.867972 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2"} err="failed to get container status \"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2\": rpc error: code = NotFound desc = could not find container \"84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2\": container with ID starting with 84e01bb777bcc1f88a06411f10b0606e739422062558e2bbee040071f6a534b2 not found: ID does not exist" Jul 16 21:02:44.872191 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.872129 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:02:44.879762 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:44.879723 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-25df27a3-openshift-default-7d786d77c9-mv9mz"] Jul 16 21:02:45.099628 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.099539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:45.232629 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.232596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:45.403592 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.403494 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:02:45.404003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.403981 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:02:45.404003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.403996 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:02:45.404099 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.404008 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:02:45.404099 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.404013 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:02:45.404099 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.404071 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:02:45.407331 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.407287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.412596 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.412559 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f05f95fc-openshift-default-dockercfg-54dv6\"" Jul 16 21:02:45.441821 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.441792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:45.461556 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.461522 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:02:45.506611 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcjq\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.506834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.507043 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.507043 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.506926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.607846 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.607806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.607846 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.607850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.607875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608230 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608333 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608399 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcjq\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608399 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608502 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608502 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608475 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608612 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608753 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608818 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.608818 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.608788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.610447 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.610422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.610897 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.610881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.616787 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.616750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.616925 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.616902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcjq\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq\") pod \"e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.719532 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.719487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:45.872367 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.872332 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:02:45.874892 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:02:45.874861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afb0cf1_7000_49ec_9b41_5458aaf6a5cd.slice/crio-f0be696b66b8b1dbac1dc70b23c656155e6cd8d00e27f59ed764949603241482 WatchSource:0}: Error finding container f0be696b66b8b1dbac1dc70b23c656155e6cd8d00e27f59ed764949603241482: Status 404 returned error can't find the container with id f0be696b66b8b1dbac1dc70b23c656155e6cd8d00e27f59ed764949603241482 Jul 16 21:02:45.877098 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.877065 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:45.877199 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.877148 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:45.877261 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:45.877214 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:46.000823 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.000739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:46.040617 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.040582 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" path="/var/lib/kubelet/pods/f5398fdd-ce1a-428b-ab6b-e902f65c6857/volumes" Jul 16 21:02:46.057474 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.057441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:46.570179 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.570143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:46.680080 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.680041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:46.782078 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.782039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:46.860823 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.860731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerStarted","Data":"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5"} Jul 16 21:02:46.860823 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.860786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerStarted","Data":"f0be696b66b8b1dbac1dc70b23c656155e6cd8d00e27f59ed764949603241482"} Jul 16 21:02:46.884622 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.884565 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podStartSLOduration=1.884525606 podStartE2EDuration="1.884525606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:02:46.883654388 +0000 UTC m=+1765.384735114" watchObservedRunningTime="2026-07-16 21:02:46.884525606 +0000 UTC m=+1765.385606332" Jul 16 21:02:46.901146 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:46.901110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:47.276663 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.276629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:47.375620 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.375582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:47.720358 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.720328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:47.725424 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.725393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:47.866453 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.866422 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:47.867417 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:47.867397 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:49.526067 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:49.526026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:02:57.931558 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.931524 2564 generic.go:356] "Generic (PLEG): container finished" podID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerID="99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5" exitCode=0 Jul 16 21:02:57.932007 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.931592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerDied","Data":"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5"} Jul 16 21:02:57.932050 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.932027 2564 scope.go:117] "RemoveContainer" containerID="99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5" Jul 16 21:02:57.933071 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.933041 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:57.933189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.933114 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:57.933189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:57.933156 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:02:58.939142 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:58.939100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerStarted","Data":"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5"} Jul 16 21:02:59.720731 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:59.720688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:02:59.721832 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:59.721805 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:02:59.721968 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:02:59.721866 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:00.719918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:00.719877 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:00.720438 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:00.719946 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:01.720589 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:01.720543 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:01.720987 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:01.720604 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:02.720163 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:02.720115 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:02.720408 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:02.720197 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:02.867103 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:02.867060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:03:03.720714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:03.720673 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:03.720986 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:03.720728 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:04.720196 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:04.720148 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:04.720631 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:04.720220 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:05.720790 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:05.720750 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 21:03:05.721175 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:05.720812 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 21:03:06.723841 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:06.723805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:03:06.724852 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:06.724828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:03:22.049381 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:22.049355 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:03:22.053920 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:03:22.053891 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:04:01.030460 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:01.030419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:06.218018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.217982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:06.362968 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.362930 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:04:06.363185 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.363162 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" containerID="cri-o://eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5" gracePeriod=30 Jul 16 21:04:06.394122 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.394091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:06.500463 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.500428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:06.726647 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.726606 2564 patch_prober.go:28] interesting pod/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:04:06.726834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:06.726674 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:04:11.619648 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.619621 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:04:11.650428 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650282 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.650428 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650368 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.650428 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650404 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhcjq\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.650712 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650555 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:04:11.650854 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.650854 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650833 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:04:11.651034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.651034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650891 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.651034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.651034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.650989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.651239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.651055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs\") pod \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\" (UID: \"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd\") " Jul 16 21:04:11.652003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.651413 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.652003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.651437 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.652003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.651819 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:04:11.652003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.651886 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:04:11.652955 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.652917 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:04:11.653567 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.653543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:04:11.653676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.653642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq" (OuterVolumeSpecName: "kube-api-access-rhcjq") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "kube-api-access-rhcjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:04:11.653754 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.653718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:04:11.654137 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.654114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" (UID: "4afb0cf1-7000-49ec-9b41-5458aaf6a5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:04:11.752793 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752752 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.752793 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752788 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcjq\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-kube-api-access-rhcjq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.753005 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752803 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.753005 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752815 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.753005 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752827 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.753005 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752838 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:11.753005 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:11.752849 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:04:12.322054 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.321963 2564 generic.go:356] "Generic (PLEG): container finished" podID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerID="eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5" exitCode=0 Jul 16 21:04:12.322054 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.322014 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" Jul 16 21:04:12.322054 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.322038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerDied","Data":"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5"} Jul 16 21:04:12.322296 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.322092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx" event={"ID":"4afb0cf1-7000-49ec-9b41-5458aaf6a5cd","Type":"ContainerDied","Data":"f0be696b66b8b1dbac1dc70b23c656155e6cd8d00e27f59ed764949603241482"} Jul 16 21:04:12.322296 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.322116 2564 scope.go:117] "RemoveContainer" containerID="eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5" Jul 16 21:04:12.331043 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.331026 2564 scope.go:117] "RemoveContainer" containerID="99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5" Jul 16 21:04:12.339241 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.339219 2564 scope.go:117] "RemoveContainer" containerID="eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5" Jul 16 21:04:12.339558 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:04:12.339538 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5\": container with ID starting with eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5 not found: ID does not exist" containerID="eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5" Jul 16 21:04:12.339627 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.339570 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5"} err="failed to get container status \"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5\": rpc error: code = NotFound desc = could not find container \"eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5\": container with ID starting with eb8005410239c2f509391a01339ced56c737e8727e1ca6492bb159feda39d2b5 not found: ID does not exist" Jul 16 21:04:12.339627 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.339591 2564 scope.go:117] "RemoveContainer" containerID="99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5" Jul 16 21:04:12.339836 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:04:12.339821 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5\": container with ID starting with 99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5 not found: ID does not exist" containerID="99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5" Jul 16 21:04:12.339887 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.339840 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5"} err="failed to get container status \"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5\": rpc error: code = NotFound desc = could not find container \"99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5\": container with ID starting with 99ec465213a38d8c37986895cf4623e5e47cd348fae96c5453a56c34484185d5 not found: ID does not exist" Jul 16 21:04:12.343774 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.343750 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:04:12.354132 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.354107 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f05f95fc-openshift-default-7d4c6fdb87-ksndx"] Jul 16 21:04:12.577809 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.577722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:12.646483 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:12.646452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:13.165837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:13.165803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:13.394366 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:13.394330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:14.040623 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:14.040579 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" path="/var/lib/kubelet/pods/4afb0cf1-7000-49ec-9b41-5458aaf6a5cd/volumes" Jul 16 21:04:15.103613 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:15.103580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:18.113409 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:18.113353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:24.888692 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:24.888648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:25.170260 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:25.170173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:25.337917 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:25.337885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:25.709518 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:25.709450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:51.244498 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:51.244413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:51.345112 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:51.345080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:51.640242 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:51.640148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:04:58.286379 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:04:58.286337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:00.638049 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:00.637987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:14.618988 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:14.618951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:20.349024 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:20.348990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:20.454497 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:20.454424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:20.550926 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:20.550896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:31.134775 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:31.134742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:33.831537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:33.831499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:38.812972 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:38.812935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:38.912131 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:38.912104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:39.710608 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:39.710569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:39.804960 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:39.804931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:39.905367 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:39.905331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:41.556003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:41.555958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:47.876411 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:47.876373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:47.971932 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:47.971897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:48.555488 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:48.555442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:54.484599 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:54.484560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:54.584738 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:54.584702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:54.767626 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:54.767549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:55.035009 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.034934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:05:55.035408 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.035393 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:05:55.035467 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.035410 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:05:55.035503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.035478 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5398fdd-ce1a-428b-ab6b-e902f65c6857" containerName="istio-proxy" Jul 16 21:05:55.035503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.035488 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:05:55.035503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.035495 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:05:55.038539 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.038523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.041707 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.041686 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a3e438d9-openshift-default-dockercfg-nfnv2\"" Jul 16 21:05:55.069855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.069829 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:05:55.150768 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.150945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.150945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.151189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.151027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cb7d\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.151189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.151070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252171 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252360 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252360 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252360 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252521 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cb7d\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252753 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252814 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252873 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.252937 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.252918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.253265 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.253215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.254974 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.254949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.255087 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.255062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.261119 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.261089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.261221 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.261198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cb7d\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d\") pod \"e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.349780 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.349690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:55.518651 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.518603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:05:55.521146 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:05:55.521117 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c9e15a_5bdc_45fb_991e_1004447e2d16.slice/crio-bec41d1a9fb3b3e76f34658ac54c7688a495742e7b0525654fa06247025b2651 WatchSource:0}: Error finding container bec41d1a9fb3b3e76f34658ac54c7688a495742e7b0525654fa06247025b2651: Status 404 returned error can't find the container with id bec41d1a9fb3b3e76f34658ac54c7688a495742e7b0525654fa06247025b2651 Jul 16 21:05:55.523733 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.523697 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:05:55.523808 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.523779 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:05:55.523846 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.523820 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:05:55.863603 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.863557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerStarted","Data":"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3"} Jul 16 21:05:55.863780 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.863662 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerStarted","Data":"bec41d1a9fb3b3e76f34658ac54c7688a495742e7b0525654fa06247025b2651"} Jul 16 21:05:55.888431 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:55.888368 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podStartSLOduration=0.888352175 podStartE2EDuration="888.352175ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:05:55.886240113 +0000 UTC m=+1954.387320839" watchObservedRunningTime="2026-07-16 21:05:55.888352175 +0000 UTC m=+1954.389432900" Jul 16 21:05:56.350850 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:56.350818 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:56.355715 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:56.355690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:56.869353 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:56.869321 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:56.870420 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:56.870401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:05:56.894239 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:56.894207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:57.127323 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:57.127190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:57.992585 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:57.992533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:58.096731 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:58.096701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:58.188443 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:58.188406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:58.292233 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:58.292139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:58.948066 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:58.948034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:05:59.044055 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:05:59.044007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:02.470512 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:02.470478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:03.120826 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:03.120790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:03.220531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:03.220494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:03.572993 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:03.572953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:09.283266 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:09.283222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:09.399331 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:09.399259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:12.911577 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:12.911476 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 21:06:12.911577 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:12.911546 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 21:06:13.975711 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.975680 2564 generic.go:356] "Generic (PLEG): container finished" podID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerID="debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3" exitCode=0 Jul 16 21:06:13.976128 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.975769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerDied","Data":"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3"} Jul 16 21:06:13.976231 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.976218 2564 scope.go:117] "RemoveContainer" containerID="debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3" Jul 16 21:06:13.977263 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.977240 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:13.977343 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.977329 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:13.977383 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:13.977367 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:14.982635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:14.982593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerStarted","Data":"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1"} Jul 16 21:06:15.349989 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:15.349897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:15.350155 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:15.350119 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 21:06:15.350200 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:15.350169 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 21:06:16.351102 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:16.351069 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 21:06:16.351557 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:16.351125 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 21:06:17.351016 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:17.350972 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 21:06:17.351236 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:17.351038 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 21:06:18.351059 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:18.351014 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 21:06:18.351245 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:18.351093 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 21:06:19.350190 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:19.350152 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 21:06:19.350398 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:19.350224 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 21:06:19.722008 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:19.721974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:20.353922 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:20.353883 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:20.354313 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:20.354280 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:20.354918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:20.354900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:24.838802 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:24.838763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:25.235451 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:25.235411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:31.342693 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.342644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:31.348688 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.348658 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:06:31.348947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.348898 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" containerID="cri-o://ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1" gracePeriod=30 Jul 16 21:06:31.420481 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.420446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:31.595703 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.595616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:31.738700 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:31.738649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:33.059477 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:33.059437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:33.124701 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:33.124667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:35.356659 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:35.356619 2564 patch_prober.go:28] interesting pod/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:06:35.357029 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:35.356672 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:06:36.606232 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.606210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:36.778599 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.778787 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cb7d\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.778787 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778646 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.778787 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.778787 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.779018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.779018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.779018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778912 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:06:36.779018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778923 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.779018 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.778973 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo\") pod \"26c9e15a-5bdc-45fb-991e-1004447e2d16\" (UID: \"26c9e15a-5bdc-45fb-991e-1004447e2d16\") " Jul 16 21:06:36.779273 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779115 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:06:36.779273 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:06:36.779273 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:06:36.779492 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779470 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data" (OuterVolumeSpecName: "istio-data") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:06:36.779557 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779475 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.779698 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779618 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.779698 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779639 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.779698 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.779652 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9e15a-5bdc-45fb-991e-1004447e2d16-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.781096 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.781071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token" (OuterVolumeSpecName: "istio-token") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:06:36.781189 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.781116 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d" (OuterVolumeSpecName: "kube-api-access-2cb7d") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "kube-api-access-2cb7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:06:36.781444 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.781422 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:06:36.781519 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.781424 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26c9e15a-5bdc-45fb-991e-1004447e2d16" (UID: "26c9e15a-5bdc-45fb-991e-1004447e2d16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:06:36.886975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.881050 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.886975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.881322 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.886975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.881340 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.886975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.881362 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9e15a-5bdc-45fb-991e-1004447e2d16-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:36.886975 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:36.881379 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cb7d\" (UniqueName: \"kubernetes.io/projected/26c9e15a-5bdc-45fb-991e-1004447e2d16-kube-api-access-2cb7d\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:06:37.102354 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.102240 2564 generic.go:356] "Generic (PLEG): container finished" podID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerID="ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1" exitCode=0 Jul 16 21:06:37.102354 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.102335 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" Jul 16 21:06:37.102354 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.102341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerDied","Data":"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1"} Jul 16 21:06:37.102612 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.102395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx" event={"ID":"26c9e15a-5bdc-45fb-991e-1004447e2d16","Type":"ContainerDied","Data":"bec41d1a9fb3b3e76f34658ac54c7688a495742e7b0525654fa06247025b2651"} Jul 16 21:06:37.102612 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.102421 2564 scope.go:117] "RemoveContainer" containerID="ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1" Jul 16 21:06:37.112788 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.112764 2564 scope.go:117] "RemoveContainer" containerID="debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3" Jul 16 21:06:37.121188 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.121164 2564 scope.go:117] "RemoveContainer" containerID="ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1" Jul 16 21:06:37.121483 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:06:37.121462 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1\": container with ID starting with ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1 not found: ID does not exist" containerID="ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1" Jul 16 21:06:37.121547 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.121492 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1"} err="failed to get container status \"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1\": rpc error: code = NotFound desc = could not find container \"ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1\": container with ID starting with ca685f35d26de7ed13d9a94388ca7a4f0955eafd81bd9002e7d5e93264057cd1 not found: ID does not exist" Jul 16 21:06:37.121547 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.121511 2564 scope.go:117] "RemoveContainer" containerID="debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3" Jul 16 21:06:37.121744 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:06:37.121726 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3\": container with ID starting with debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3 not found: ID does not exist" containerID="debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3" Jul 16 21:06:37.121789 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.121750 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3"} err="failed to get container status \"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3\": rpc error: code = NotFound desc = could not find container \"debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3\": container with ID starting with debfd243d6d463a4100907cd8d9d4b97b56a608a3a4139707892cbf96b9b2ad3 not found: ID does not exist" Jul 16 21:06:37.129344 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.129319 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:06:37.132768 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.132748 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a3e438d9-openshift-default-5c66954dc9-md8fx"] Jul 16 21:06:37.759368 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.759327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:37.846257 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:37.846228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:38.040259 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:38.040175 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" path="/var/lib/kubelet/pods/26c9e15a-5bdc-45fb-991e-1004447e2d16/volumes" Jul 16 21:06:39.012174 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:39.012145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:39.081148 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:39.081092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:44.039278 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:44.039240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:44.145508 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:44.145463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:50.384237 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:50.384198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:50.487623 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:50.487587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:50.862349 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:50.862319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:51.132805 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.132719 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:06:51.133102 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133089 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:06:51.133180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133103 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afb0cf1-7000-49ec-9b41-5458aaf6a5cd" containerName="istio-proxy" Jul 16 21:06:51.133180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133115 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.133180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133120 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.133180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133130 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.133180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.133450 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133216 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.133450 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.133225 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c9e15a-5bdc-45fb-991e-1004447e2d16" containerName="istio-proxy" Jul 16 21:06:51.137951 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.137926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.140985 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.140961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27a1e9ed-openshift-default-dockercfg-pvsr5\"" Jul 16 21:06:51.154811 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.154786 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:06:51.170136 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.170108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:51.212727 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.212727 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.212930 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.212930 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.213021 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.213021 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.212995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rh4\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.213021 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.213016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.213126 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.213034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.213126 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.213061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314203 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rh4\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314203 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314900 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314900 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.314993 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.314908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.315093 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.315071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.315687 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.315664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.316959 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.316938 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.317162 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.317145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.323413 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.323385 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rh4\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.323510 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.323469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token\") pod \"e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.449809 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.449727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:51.597042 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.597010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:06:51.598695 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:06:51.598665 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a797a9_318b_4aee_ab6e_5cfa1651c37e.slice/crio-8c27e922d0e4971360ffc72f422b5d86095220f04cd437fde4d9934762262a69 WatchSource:0}: Error finding container 8c27e922d0e4971360ffc72f422b5d86095220f04cd437fde4d9934762262a69: Status 404 returned error can't find the container with id 8c27e922d0e4971360ffc72f422b5d86095220f04cd437fde4d9934762262a69 Jul 16 21:06:51.600944 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.600907 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:51.601025 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.601000 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:51.601065 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.601040 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:06:51.721368 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:51.721328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:52.179038 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:52.178947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerStarted","Data":"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da"} Jul 16 21:06:52.179038 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:52.179004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerStarted","Data":"8c27e922d0e4971360ffc72f422b5d86095220f04cd437fde4d9934762262a69"} Jul 16 21:06:52.203001 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:52.202952 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podStartSLOduration=1.2029370400000001 podStartE2EDuration="1.20293704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:06:52.200811908 +0000 UTC m=+2010.701892636" watchObservedRunningTime="2026-07-16 21:06:52.20293704 +0000 UTC m=+2010.704017765" Jul 16 21:06:52.450324 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:52.450207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:52.456580 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:52.456553 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:53.187771 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:53.187737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:53.188797 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:53.188775 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:06:54.188997 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:54.188966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:54.291407 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:54.291354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:54.831872 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:54.831838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:54.934091 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:54.934058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:06:58.658577 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:06:58.658538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:05.606543 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:05.606508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:07.260928 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.260900 2564 generic.go:356] "Generic (PLEG): container finished" podID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerID="7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da" exitCode=0 Jul 16 21:07:07.261286 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.260973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerDied","Data":"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da"} Jul 16 21:07:07.261411 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.261396 2564 scope.go:117] "RemoveContainer" containerID="7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da" Jul 16 21:07:07.262375 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.262348 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:07:07.262436 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.262418 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:07:07.262471 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:07.262450 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:07:08.188583 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:08.188547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:07:08.267740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:08.267704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerStarted","Data":"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506"} Jul 16 21:07:08.450869 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:08.450781 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:07:08.451804 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:08.451779 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 21:07:08.451916 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:08.451829 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 21:07:09.450412 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:09.450373 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 21:07:09.450803 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:09.450434 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 21:07:10.450159 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:10.450110 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 21:07:10.450377 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:10.450178 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 21:07:11.450840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:11.450794 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 21:07:11.451356 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:11.450875 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 21:07:12.450866 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:12.450825 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 21:07:12.451263 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:12.450880 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 21:07:13.454873 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:13.454835 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:07:13.455328 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:13.455143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:07:13.456146 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:13.456125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:07:42.341593 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:42.341501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:47.195205 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:47.195166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:49.515978 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:49.515937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:54.630711 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:54.630673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:07:55.967240 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:07:55.967205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:22.080962 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:22.080930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:08:22.087535 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:22.087509 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:08:31.801097 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:31.801063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:32.443160 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:32.443101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:38.211930 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.211888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:38.310733 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.310695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:38.492375 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.492337 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:08:38.556028 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.555998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:38.741370 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.741267 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" containerID="cri-o://5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506" gracePeriod=30 Jul 16 21:08:38.916074 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:38.915993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:39.985830 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:39.985791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:40.142180 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:40.142143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:41.055039 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:41.055003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:41.189841 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:41.189799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:42.777236 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:42.777198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:43.458376 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:43.458340 2564 patch_prober.go:28] interesting pod/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 21:08:43.458552 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:43.458411 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:08:44.012001 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.011975 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:08:44.183125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183012 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183063 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183203 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8rh4\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183245 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183287 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183339 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183369 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183711 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183405 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token\") pod \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\" (UID: \"21a797a9-318b-4aee-ab6e-5cfa1651c37e\") " Jul 16 21:08:44.183711 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183418 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:08:44.183711 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.183684 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.184524 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.184175 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:08:44.184524 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.184326 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:08:44.184524 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.184479 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:08:44.184898 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.184876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data" (OuterVolumeSpecName: "istio-data") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:08:44.186003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.185981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:08:44.186537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.186513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token" (OuterVolumeSpecName: "istio-token") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:08:44.186623 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.186568 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4" (OuterVolumeSpecName: "kube-api-access-l8rh4") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "kube-api-access-l8rh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:08:44.186740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.186723 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21a797a9-318b-4aee-ab6e-5cfa1651c37e" (UID: "21a797a9-318b-4aee-ab6e-5cfa1651c37e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:08:44.284794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284758 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8rh4\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-kube-api-access-l8rh4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.284794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284791 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.284794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284800 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.285027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284812 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.285027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284821 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.285027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284829 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.285027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284836 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.285027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.284843 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a797a9-318b-4aee-ab6e-5cfa1651c37e-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:08:44.781282 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.781252 2564 generic.go:356] "Generic (PLEG): container finished" podID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerID="5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506" exitCode=0 Jul 16 21:08:44.781529 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.781330 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" Jul 16 21:08:44.781529 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.781345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerDied","Data":"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506"} Jul 16 21:08:44.781529 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.781404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd" event={"ID":"21a797a9-318b-4aee-ab6e-5cfa1651c37e","Type":"ContainerDied","Data":"8c27e922d0e4971360ffc72f422b5d86095220f04cd437fde4d9934762262a69"} Jul 16 21:08:44.781529 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.781427 2564 scope.go:117] "RemoveContainer" containerID="5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506" Jul 16 21:08:44.798929 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.793544 2564 scope.go:117] "RemoveContainer" containerID="7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da" Jul 16 21:08:44.809150 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.809123 2564 scope.go:117] "RemoveContainer" containerID="5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506" Jul 16 21:08:44.809546 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:08:44.809524 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506\": container with ID starting with 5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506 not found: ID does not exist" containerID="5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506" Jul 16 21:08:44.809612 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.809554 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506"} err="failed to get container status \"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506\": rpc error: code = NotFound desc = could not find container \"5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506\": container with ID starting with 5a83e3640cb8932e40269117fc0c60bbb6d306ca78d54e5cce80ea63969c9506 not found: ID does not exist" Jul 16 21:08:44.809612 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.809573 2564 scope.go:117] "RemoveContainer" containerID="7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da" Jul 16 21:08:44.809844 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:08:44.809823 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da\": container with ID starting with 7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da not found: ID does not exist" containerID="7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da" Jul 16 21:08:44.809908 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.809855 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da"} err="failed to get container status \"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da\": rpc error: code = NotFound desc = could not find container \"7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da\": container with ID starting with 7699fa81ebf4023c7818d5ebb531269f7c71a1b00d14cb992fba0d69ab23e6da not found: ID does not exist" Jul 16 21:08:44.815679 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.815640 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:08:44.830143 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:44.830098 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27a1e9ed-openshift-default-748885cbf9-542fd"] Jul 16 21:08:46.040267 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:46.040234 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" path="/var/lib/kubelet/pods/21a797a9-318b-4aee-ab6e-5cfa1651c37e/volumes" Jul 16 21:08:49.261312 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:49.261275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:49.358474 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:49.358434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:49.994801 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:49.994772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:57.177184 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:57.177154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:57.283513 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:57.283476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:08:57.442537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:08:57.442446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:07.949485 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:07.949447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:08.295749 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:08.295709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:29.512567 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:29.512536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:36.407915 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:36.407881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:36.493254 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:36.493219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:36.761689 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:36.761654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:37.050509 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.050425 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:09:37.050871 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.050856 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:09:37.050926 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.050874 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:09:37.050976 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.050967 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:09:37.051011 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.050980 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:09:37.053941 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.053926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.057037 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.057015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60ce31-openshift-default-dockercfg-kzkf5\"" Jul 16 21:09:37.068430 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.068407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:09:37.081867 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.081840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:37.094741 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.094696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.094923 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.094781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.094923 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.094829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.094923 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.094875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.095091 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.094921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.095091 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.095013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.095091 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.095062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhsm\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.095240 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.095096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.095240 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.095153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195659 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhsm\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.195855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.195871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.196103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.196114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.196166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.196409 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.196684 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.196661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.198929 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.198900 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.199652 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.199630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.208924 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.208898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.209453 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.209432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhsm\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm\") pod \"e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.364433 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.364348 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:37.509357 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.509333 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:09:37.511254 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:09:37.511218 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442305c8_c248_4df0_9a00_0867ad9c1d7d.slice/crio-b9cd2a316fcc4631e954af40f427de148d8072fbffe9a8df80db53751fd29a2b WatchSource:0}: Error finding container b9cd2a316fcc4631e954af40f427de148d8072fbffe9a8df80db53751fd29a2b: Status 404 returned error can't find the container with id b9cd2a316fcc4631e954af40f427de148d8072fbffe9a8df80db53751fd29a2b Jul 16 21:09:37.513276 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.513245 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:37.513378 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.513343 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:37.513421 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.513392 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:37.968899 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:37.968867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:38.047761 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:38.047729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerStarted","Data":"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468"} Jul 16 21:09:38.047918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:38.047774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerStarted","Data":"b9cd2a316fcc4631e954af40f427de148d8072fbffe9a8df80db53751fd29a2b"} Jul 16 21:09:38.069112 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:38.069059 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podStartSLOduration=1.069044405 podStartE2EDuration="1.069044405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:09:38.067777704 +0000 UTC m=+2176.568858422" watchObservedRunningTime="2026-07-16 21:09:38.069044405 +0000 UTC m=+2176.570125131" Jul 16 21:09:38.364630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:38.364547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:38.369564 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:38.369538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:39.053335 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:39.053292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:39.054506 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:39.054488 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:40.056509 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.056474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:40.151210 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.151156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:40.583592 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.583556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:40.741264 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.741220 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:40.860429 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.860337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:40.962856 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:40.962822 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:44.726751 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:44.726713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:44.906620 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:44.906584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:45.290367 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:45.290337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:45.564944 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:45.564829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:51.136269 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:51.136223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:51.235548 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:51.235517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:51.345534 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:51.345491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:51.804785 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:51.804730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:51.904854 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:51.904800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:52.106088 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:52.105996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:55.095494 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:55.095451 2564 patch_prober.go:28] interesting pod/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 21:09:55.095866 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:55.095534 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 21:09:55.644978 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:55.644943 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:09:56.148785 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.148701 2564 generic.go:356] "Generic (PLEG): container finished" podID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerID="f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468" exitCode=0 Jul 16 21:09:56.148785 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.148769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerDied","Data":"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468"} Jul 16 21:09:56.149230 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.149204 2564 scope.go:117] "RemoveContainer" containerID="f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468" Jul 16 21:09:56.150284 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.150254 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:56.150425 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.150344 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:56.150425 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:56.150379 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 21:09:57.156374 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:57.156339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerStarted","Data":"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65"} Jul 16 21:09:57.365351 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:57.365296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:09:57.365576 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:57.365550 2564 patch_prober.go:28] interesting pod/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 21:09:57.365625 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:57.365606 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 21:09:58.365409 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:58.365368 2564 patch_prober.go:28] interesting pod/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 21:09:58.365818 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:58.365425 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 21:09:59.365453 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:59.365411 2564 patch_prober.go:28] interesting pod/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 21:09:59.365843 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:59.365469 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 21:09:59.717855 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:09:59.717766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:00.365095 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:00.365062 2564 patch_prober.go:28] interesting pod/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 16 21:10:00.365281 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:00.365117 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 16 21:10:01.369526 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:01.369492 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:10:01.370131 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:01.369900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:10:01.370729 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:01.370708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:10:01.924228 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:01.924194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:04.852597 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:04.852564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:12.619387 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:12.619353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:12.722072 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:12.722037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:24.182269 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.182235 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:10:24.182676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.182662 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:10:24.182719 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.182678 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a797a9-318b-4aee-ab6e-5cfa1651c37e" containerName="istio-proxy" Jul 16 21:10:24.184956 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.184940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.189061 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.189038 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-054dd3\"/\"kube-root-ca.crt\"" Jul 16 21:10:24.189192 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.189040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-054dd3\"/\"openshift-service-ca.crt\"" Jul 16 21:10:24.189192 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.189040 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-054dd3\"/\"test-model-054dd3-kserve-self-signed-certs\"" Jul 16 21:10:24.189192 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.189151 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-054dd3\"/\"default-dockercfg-2rxtb\"" Jul 16 21:10:24.198326 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.198281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:10:24.208512 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.208487 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:24.253450 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.253639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxgt\" (UniqueName: \"kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.253639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.253639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.253779 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.253779 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.253714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354339 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxgt\" (UniqueName: \"kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354744 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354810 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354742 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354810 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.354955 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.354935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.356918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.356898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.357034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.357018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.362794 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.362771 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxgt\" (UniqueName: \"kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt\") pod \"test-model-054dd3-kserve-798bc596f-5kfff\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.495096 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.495054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:24.640603 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:24.640550 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:10:24.642792 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:10:24.642764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d498b65_809c_4671_8c4d_dda9418b0736.slice/crio-ae337f4c9a18c870bb5bedf67b23db8d4d23da487fb17535fd1292b3d4080b58 WatchSource:0}: Error finding container ae337f4c9a18c870bb5bedf67b23db8d4d23da487fb17535fd1292b3d4080b58: Status 404 returned error can't find the container with id ae337f4c9a18c870bb5bedf67b23db8d4d23da487fb17535fd1292b3d4080b58 Jul 16 21:10:25.302772 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:25.302728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" event={"ID":"8d498b65-809c-4671-8c4d-dda9418b0736","Type":"ContainerStarted","Data":"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895"} Jul 16 21:10:25.303175 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:25.302780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" event={"ID":"8d498b65-809c-4671-8c4d-dda9418b0736","Type":"ContainerStarted","Data":"ae337f4c9a18c870bb5bedf67b23db8d4d23da487fb17535fd1292b3d4080b58"} Jul 16 21:10:25.323618 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:25.323561 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" podStartSLOduration=1.3235428040000001 podStartE2EDuration="1.323542804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:10:25.322442554 +0000 UTC m=+2223.823523270" watchObservedRunningTime="2026-07-16 21:10:25.323542804 +0000 UTC m=+2223.824623529" Jul 16 21:10:34.495461 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:34.495416 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:34.495972 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:34.495475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:34.508646 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:34.508617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:35.365390 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:35.365361 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:10:43.984969 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:43.984893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:44.849916 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:44.849884 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:10:44.950885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:44.950854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:55.078565 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:55.078536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:10:59.870595 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:10:59.870554 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" podUID="8d498b65-809c-4671-8c4d-dda9418b0736" containerName="main" containerID="cri-o://eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895" gracePeriod=45 Jul 16 21:11:00.127813 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.127746 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:11:00.194028 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.193995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:00.203484 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203449 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203624 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203528 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203624 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203568 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnxgt\" (UniqueName: \"kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203734 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203734 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203693 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203734 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm\") pod \"8d498b65-809c-4671-8c4d-dda9418b0736\" (UID: \"8d498b65-809c-4671-8c4d-dda9418b0736\") " Jul 16 21:11:00.203885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203783 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home" (OuterVolumeSpecName: "home") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:00.203934 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203905 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:00.203976 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.203937 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:00.204103 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.204086 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.204176 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.204120 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.204176 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.204129 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.205947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.205924 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm" (OuterVolumeSpecName: "dshm") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:00.205947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.205918 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:11:00.206171 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.205947 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt" (OuterVolumeSpecName: "kube-api-access-tnxgt") pod "8d498b65-809c-4671-8c4d-dda9418b0736" (UID: "8d498b65-809c-4671-8c4d-dda9418b0736"). InnerVolumeSpecName "kube-api-access-tnxgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:11:00.304953 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.304916 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnxgt\" (UniqueName: \"kubernetes.io/projected/8d498b65-809c-4671-8c4d-dda9418b0736-kube-api-access-tnxgt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.304953 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.304946 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d498b65-809c-4671-8c4d-dda9418b0736-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.304953 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.304957 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d498b65-809c-4671-8c4d-dda9418b0736-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:00.488688 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.488652 2564 generic.go:356] "Generic (PLEG): container finished" podID="8d498b65-809c-4671-8c4d-dda9418b0736" containerID="eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895" exitCode=0 Jul 16 21:11:00.488884 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.488700 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" Jul 16 21:11:00.488884 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.488734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" event={"ID":"8d498b65-809c-4671-8c4d-dda9418b0736","Type":"ContainerDied","Data":"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895"} Jul 16 21:11:00.488884 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.488778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff" event={"ID":"8d498b65-809c-4671-8c4d-dda9418b0736","Type":"ContainerDied","Data":"ae337f4c9a18c870bb5bedf67b23db8d4d23da487fb17535fd1292b3d4080b58"} Jul 16 21:11:00.488884 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.488796 2564 scope.go:117] "RemoveContainer" containerID="eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895" Jul 16 21:11:00.498669 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.498651 2564 scope.go:117] "RemoveContainer" containerID="eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895" Jul 16 21:11:00.498947 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:11:00.498930 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895\": container with ID starting with eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895 not found: ID does not exist" containerID="eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895" Jul 16 21:11:00.498994 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.498956 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895"} err="failed to get container status \"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895\": rpc error: code = NotFound desc = could not find container \"eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895\": container with ID starting with eb58af28eeb04ff61f7683831571d5ddf50eb0b80a57cbbebd63ac7a49cc5895 not found: ID does not exist" Jul 16 21:11:00.514946 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.514921 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:11:00.518322 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.518284 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-054dd3/test-model-054dd3-kserve-798bc596f-5kfff"] Jul 16 21:11:00.619725 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:00.619690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:02.039921 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:02.039885 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d498b65-809c-4671-8c4d-dda9418b0736" path="/var/lib/kubelet/pods/8d498b65-809c-4671-8c4d-dda9418b0736/volumes" Jul 16 21:11:06.748253 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:06.748219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:06.848032 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:06.847975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:07.011917 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:07.011832 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:11:07.012095 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:07.012072 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" containerID="cri-o://71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65" gracePeriod=30 Jul 16 21:11:07.034630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:07.034593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:07.323464 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:07.323356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:12.281185 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.281117 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:11:12.427206 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427256 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427388 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427639 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvhsm\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm\") pod \"442305c8-c248-4df0-9a00-0867ad9c1d7d\" (UID: \"442305c8-c248-4df0-9a00-0867ad9c1d7d\") " Jul 16 21:11:12.427786 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:12.427835 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427783 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:11:12.427883 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427842 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.428001 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.427908 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:12.428084 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.428044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:12.428175 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.428152 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:12.429890 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.429860 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 21:11:12.430019 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.429893 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm" (OuterVolumeSpecName: "kube-api-access-bvhsm") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "kube-api-access-bvhsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:11:12.430130 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.430097 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:11:12.430211 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.430194 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "442305c8-c248-4df0-9a00-0867ad9c1d7d" (UID: "442305c8-c248-4df0-9a00-0867ad9c1d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:11:12.528434 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528404 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528434 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528430 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442305c8-c248-4df0-9a00-0867ad9c1d7d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528434 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528440 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528448 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhsm\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-kube-api-access-bvhsm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528462 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528474 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528489 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.528714 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.528502 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442305c8-c248-4df0-9a00-0867ad9c1d7d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:11:12.549295 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.549207 2564 generic.go:356] "Generic (PLEG): container finished" podID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerID="71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65" exitCode=0 Jul 16 21:11:12.549295 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.549255 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" Jul 16 21:11:12.549487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.549323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerDied","Data":"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65"} Jul 16 21:11:12.549487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.549384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m" event={"ID":"442305c8-c248-4df0-9a00-0867ad9c1d7d","Type":"ContainerDied","Data":"b9cd2a316fcc4631e954af40f427de148d8072fbffe9a8df80db53751fd29a2b"} Jul 16 21:11:12.549487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.549406 2564 scope.go:117] "RemoveContainer" containerID="71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65" Jul 16 21:11:12.558507 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.558488 2564 scope.go:117] "RemoveContainer" containerID="f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468" Jul 16 21:11:12.568470 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.568451 2564 scope.go:117] "RemoveContainer" containerID="71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65" Jul 16 21:11:12.568853 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:11:12.568830 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65\": container with ID starting with 71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65 not found: ID does not exist" containerID="71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65" Jul 16 21:11:12.568925 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.568865 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65"} err="failed to get container status \"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65\": rpc error: code = NotFound desc = could not find container \"71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65\": container with ID starting with 71d5b98c8aebcb7e225073f9685b56180b5c547425bc03acde18052c9c44ac65 not found: ID does not exist" Jul 16 21:11:12.568925 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.568884 2564 scope.go:117] "RemoveContainer" containerID="f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468" Jul 16 21:11:12.569176 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:11:12.569157 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468\": container with ID starting with f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468 not found: ID does not exist" containerID="f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468" Jul 16 21:11:12.569220 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.569182 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468"} err="failed to get container status \"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468\": rpc error: code = NotFound desc = could not find container \"f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468\": container with ID starting with f3f3188e565a1ccd1a30c254d24d59297f23fe53b697a7124a46760ddd63a468 not found: ID does not exist" Jul 16 21:11:12.572364 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.572343 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:11:12.582798 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:12.582774 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60ce31-openshift-default-74ffb67bf7-pvs2m"] Jul 16 21:11:14.044516 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:14.042078 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" path="/var/lib/kubelet/pods/442305c8-c248-4df0-9a00-0867ad9c1d7d/volumes" Jul 16 21:11:17.472725 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:17.472682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:22.588556 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:22.588498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:22.773417 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:22.773372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:23.008073 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:23.008019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:29.004024 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:29.003989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:29.104618 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:29.104588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:29.278571 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:29.278476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:29.527493 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:29.527462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:29.883695 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:29.883654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:30.227667 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:30.227631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:30.430489 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:30.430453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:32.778249 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:32.778213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:32.886977 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:32.886945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:33.010440 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:33.010410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:33.350724 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:33.350690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:33.832235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:33.832204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:33.927428 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:33.927391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:37.255497 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:37.255454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:37.738136 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:37.738100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:38.077079 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:38.076990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:38.177751 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:38.177713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:40.773873 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:40.773843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:40.856251 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:40.856221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:40.943261 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:40.943226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:41.448716 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:41.448676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:41.548776 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:41.548746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:41.649031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:41.648997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:45.299262 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:45.299230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:51.208144 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:51.208105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:11:51.309205 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:11:51.309176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:02.247588 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:02.247546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:02.358161 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:02.358127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:08.521926 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:08.521895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:13.621694 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:13.621617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:14.013423 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:14.013382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:19.717290 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:19.717228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:19.816960 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:19.816928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:19.996127 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:19.996095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:20.239955 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.239924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:20.350675 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.350597 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m"] Jul 16 21:12:20.350973 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.350961 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d498b65-809c-4671-8c4d-dda9418b0736" containerName="main" Jul 16 21:12:20.351014 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.350974 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d498b65-809c-4671-8c4d-dda9418b0736" containerName="main" Jul 16 21:12:20.351014 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351000 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:12:20.351014 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351006 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:12:20.351014 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351014 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:12:20.351143 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351020 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:12:20.351143 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351070 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:12:20.351143 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.351079 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d498b65-809c-4671-8c4d-dda9418b0736" containerName="main" Jul 16 21:12:20.353996 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.353980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:20.356633 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.356615 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:12:20.356717 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.356657 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fcvh8\"" Jul 16 21:12:20.362034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.362012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m"] Jul 16 21:12:20.370505 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.370482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a896c7-9558m\" (UID: \"f6de7096-74d6-4c10-b5cc-fe5a233f81e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:20.471829 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.471793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a896c7-9558m\" (UID: \"f6de7096-74d6-4c10-b5cc-fe5a233f81e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:20.472565 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.472541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a896c7-9558m\" (UID: \"f6de7096-74d6-4c10-b5cc-fe5a233f81e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:20.667066 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.666970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:20.808720 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.808696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m"] Jul 16 21:12:20.810996 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:12:20.810967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6de7096_74d6_4c10_b5cc_fe5a233f81e2.slice/crio-7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1 WatchSource:0}: Error finding container 7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1: Status 404 returned error can't find the container with id 7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1 Jul 16 21:12:20.812893 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.812877 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:12:20.904413 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:20.904373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" event={"ID":"f6de7096-74d6-4c10-b5cc-fe5a233f81e2","Type":"ContainerStarted","Data":"7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1"} Jul 16 21:12:23.921774 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:23.921735 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6de7096-74d6-4c10-b5cc-fe5a233f81e2" containerID="21a2a5c8d69f79efac7b2ed1c2dc9f05c78b601ff3cf147a8b9ac3570c632af8" exitCode=0 Jul 16 21:12:23.922185 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:23.921822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" event={"ID":"f6de7096-74d6-4c10-b5cc-fe5a233f81e2","Type":"ContainerDied","Data":"21a2a5c8d69f79efac7b2ed1c2dc9f05c78b601ff3cf147a8b9ac3570c632af8"} Jul 16 21:12:25.067432 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.067405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:25.111071 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.111034 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca\") pod \"f6de7096-74d6-4c10-b5cc-fe5a233f81e2\" (UID: \"f6de7096-74d6-4c10-b5cc-fe5a233f81e2\") " Jul 16 21:12:25.111438 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.111414 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6de7096-74d6-4c10-b5cc-fe5a233f81e2" (UID: "f6de7096-74d6-4c10-b5cc-fe5a233f81e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:12:25.212815 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.212726 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6de7096-74d6-4c10-b5cc-fe5a233f81e2-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:12:25.937099 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.937072 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" Jul 16 21:12:25.937321 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.937119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a896c7-9558m" event={"ID":"f6de7096-74d6-4c10-b5cc-fe5a233f81e2","Type":"ContainerDied","Data":"7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1"} Jul 16 21:12:25.937321 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:25.937172 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7efcc06e35206545ee5d37a235ba15154b8333b25e901c4f7c117c29354a39c1" Jul 16 21:12:35.425454 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:35.425418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:40.540737 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:40.540689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:40.946359 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:40.946243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:46.611083 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:46.611031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:46.672390 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:46.672356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:46.889243 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:46.889162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:47.158991 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:47.158893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:47.487261 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:47.487226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:47.850526 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:47.850417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:48.210172 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:48.210076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:51.288932 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:51.288897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:51.390200 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:51.390159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:51.502800 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:51.502765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:52.051647 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:52.051610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:52.142925 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:52.142891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:55.814596 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:55.814557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:56.159480 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:56.159366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:56.507680 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:56.507638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:56.709031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:56.708993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:59.342287 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:59.342250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:59.438257 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:59.438218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:59.535437 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:59.535403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:12:59.769315 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:12:59.769277 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:00.134135 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:00.134041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:00.230003 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:00.229960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:03.860780 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:03.860747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:09.957968 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:09.957937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:10.011981 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:10.011940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:12.128676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:12.128629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:12.887058 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:12.887028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:12.986422 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:12.986390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:14.198882 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.198842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:13:14.199472 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.199425 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6de7096-74d6-4c10-b5cc-fe5a233f81e2" containerName="revoke-keys" Jul 16 21:13:14.199472 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.199473 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6de7096-74d6-4c10-b5cc-fe5a233f81e2" containerName="revoke-keys" Jul 16 21:13:14.199632 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.199595 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6de7096-74d6-4c10-b5cc-fe5a233f81e2" containerName="revoke-keys" Jul 16 21:13:14.199632 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.199611 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="442305c8-c248-4df0-9a00-0867ad9c1d7d" containerName="istio-proxy" Jul 16 21:13:14.203872 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.203851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.208097 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.208074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"auth-test-model-09402f-kserve-self-signed-certs\"" Jul 16 21:13:14.208235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.208105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"openshift-service-ca.crt\"" Jul 16 21:13:14.208235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.208163 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"kube-root-ca.crt\"" Jul 16 21:13:14.208235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.208170 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"default-dockercfg-pd67w\"" Jul 16 21:13:14.214781 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.214757 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:13:14.228441 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.228410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:14.266759 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.266918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.266918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.266918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mzd\" (UniqueName: \"kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.267052 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.267052 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.266956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367432 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mzd\" (UniqueName: \"kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.367940 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.368000 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.367964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.368128 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.368109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.370811 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.370737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.371193 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.371172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.377291 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.377269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mzd\" (UniqueName: \"kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd\") pod \"auth-test-model-09402f-kserve-54c4cf5687-8kxvv\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.514918 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.514887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:14.653987 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:14.653958 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:13:14.655985 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:13:14.655955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode80dab05_a740_455b_9826_b8b0a51b1f54.slice/crio-a709ef85d9326c42ff77a31c58e89ae289e855070733329fbd995fd78f443e92 WatchSource:0}: Error finding container a709ef85d9326c42ff77a31c58e89ae289e855070733329fbd995fd78f443e92: Status 404 returned error can't find the container with id a709ef85d9326c42ff77a31c58e89ae289e855070733329fbd995fd78f443e92 Jul 16 21:13:15.196371 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:15.196336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" event={"ID":"e80dab05-a740-455b-9826-b8b0a51b1f54","Type":"ContainerStarted","Data":"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10"} Jul 16 21:13:15.196557 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:15.196381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" event={"ID":"e80dab05-a740-455b-9826-b8b0a51b1f54","Type":"ContainerStarted","Data":"a709ef85d9326c42ff77a31c58e89ae289e855070733329fbd995fd78f443e92"} Jul 16 21:13:15.215345 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:15.215284 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" podStartSLOduration=1.21526951 podStartE2EDuration="1.21526951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:13:15.213500737 +0000 UTC m=+2393.714581499" watchObservedRunningTime="2026-07-16 21:13:15.21526951 +0000 UTC m=+2393.716350235" Jul 16 21:13:19.738578 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:19.738545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:20.861930 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:20.861883 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:20.964789 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:20.964753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:13:22.111885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:22.111856 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:13:22.119517 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:22.119496 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:13:24.515606 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:24.515559 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:24.516000 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:24.515635 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:24.528610 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:24.528582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:25.257802 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:25.257774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:13:31.248571 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:13:31.248538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:14:31.152407 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:14:31.152374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:15:00.151885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.151842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk"] Jul 16 21:15:00.155198 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.155178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" Jul 16 21:15:00.156011 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.155987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2"] Jul 16 21:15:00.157910 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.157889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fcvh8\"" Jul 16 21:15:00.159609 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.159589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" Jul 16 21:15:00.166609 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.166579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" Jul 16 21:15:00.168937 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.168906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk"] Jul 16 21:15:00.176268 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.176191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2"] Jul 16 21:15:00.176635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.176611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" Jul 16 21:15:00.328740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.328716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk"] Jul 16 21:15:00.332038 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:15:00.331975 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaab534a_3f82_455b_bb9f_255a3de57210.slice/crio-7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50 WatchSource:0}: Error finding container 7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50: Status 404 returned error can't find the container with id 7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50 Jul 16 21:15:00.351999 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.351974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2"] Jul 16 21:15:00.353506 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:15:00.353482 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825b70c1_3874_4b46_b2bf_ec0a71e1ca50.slice/crio-dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727 WatchSource:0}: Error finding container dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727: Status 404 returned error can't find the container with id dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727 Jul 16 21:15:00.753699 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.753647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" event={"ID":"caab534a-3f82-455b-bb9f-255a3de57210","Type":"ContainerStarted","Data":"5e47dc9bf2f8aea4c852afb6695b6143b0438c5c288ec7e771b95922a6b58865"} Jul 16 21:15:00.753903 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.753716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" event={"ID":"caab534a-3f82-455b-bb9f-255a3de57210","Type":"ContainerStarted","Data":"7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50"} Jul 16 21:15:00.754856 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.754827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" event={"ID":"825b70c1-3874-4b46-b2bf-ec0a71e1ca50","Type":"ContainerStarted","Data":"dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727"} Jul 16 21:15:00.782311 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:00.782250 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" podStartSLOduration=0.78223097 podStartE2EDuration="782.23097ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:00.778180034 +0000 UTC m=+2499.279260742" watchObservedRunningTime="2026-07-16 21:15:00.78223097 +0000 UTC m=+2499.283311695" Jul 16 21:15:01.761803 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:01.761701 2564 generic.go:356] "Generic (PLEG): container finished" podID="825b70c1-3874-4b46-b2bf-ec0a71e1ca50" containerID="294be5734e4f9ed916eeab83db63cc3a2f425484c42a6eaa141db5dffc518dc6" exitCode=0 Jul 16 21:15:01.762253 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:01.761789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" event={"ID":"825b70c1-3874-4b46-b2bf-ec0a71e1ca50","Type":"ContainerDied","Data":"294be5734e4f9ed916eeab83db63cc3a2f425484c42a6eaa141db5dffc518dc6"} Jul 16 21:15:01.763253 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:01.763233 2564 generic.go:356] "Generic (PLEG): container finished" podID="caab534a-3f82-455b-bb9f-255a3de57210" containerID="5e47dc9bf2f8aea4c852afb6695b6143b0438c5c288ec7e771b95922a6b58865" exitCode=0 Jul 16 21:15:01.763356 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:01.763329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" event={"ID":"caab534a-3f82-455b-bb9f-255a3de57210","Type":"ContainerDied","Data":"5e47dc9bf2f8aea4c852afb6695b6143b0438c5c288ec7e771b95922a6b58865"} Jul 16 21:15:02.926209 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:02.926178 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" Jul 16 21:15:02.929476 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:02.929453 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" Jul 16 21:15:03.775078 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.775046 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" Jul 16 21:15:03.775078 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.775061 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737275-877v2" event={"ID":"825b70c1-3874-4b46-b2bf-ec0a71e1ca50","Type":"ContainerDied","Data":"dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727"} Jul 16 21:15:03.775382 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.775123 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee8b20fd422f2b63f53c3f591d1619acbd65ad717e9fb571af79e1d50607727" Jul 16 21:15:03.776537 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.776513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" event={"ID":"caab534a-3f82-455b-bb9f-255a3de57210","Type":"ContainerDied","Data":"7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50"} Jul 16 21:15:03.776684 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.776541 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk" Jul 16 21:15:03.776684 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:03.776553 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb919ccb1bc847a83e07c35a81a8b0e3cf330d416e6b6bdc0c3dc5f95563e50" Jul 16 21:15:26.093231 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:26.093153 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:15:26.117888 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:26.117858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:15:27.758686 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:27.758645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:15:38.996131 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996094 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:15:38.996692 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996673 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caab534a-3f82-455b-bb9f-255a3de57210" containerName="cleanup" Jul 16 21:15:38.996776 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996696 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="caab534a-3f82-455b-bb9f-255a3de57210" containerName="cleanup" Jul 16 21:15:38.996776 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996737 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825b70c1-3874-4b46-b2bf-ec0a71e1ca50" containerName="cleanup" Jul 16 21:15:38.996776 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996747 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="825b70c1-3874-4b46-b2bf-ec0a71e1ca50" containerName="cleanup" Jul 16 21:15:38.996929 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996843 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="caab534a-3f82-455b-bb9f-255a3de57210" containerName="cleanup" Jul 16 21:15:38.996929 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:38.996861 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="825b70c1-3874-4b46-b2bf-ec0a71e1ca50" containerName="cleanup" Jul 16 21:15:39.000067 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.000047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.003053 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.003026 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"sub-test-model-09402f-kserve-self-signed-certs\"" Jul 16 21:15:39.009865 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.009842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:15:39.036656 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.036614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:15:39.111945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.111910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.111945 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.111945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6vb\" (UniqueName: \"kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.112170 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.111975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.112170 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.111993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.112170 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.112017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.112170 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.112075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213576 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213576 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.213837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.213742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6vb\" (UniqueName: \"kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.214051 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.214031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.214096 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.214071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.214147 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.214091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.216084 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.216062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.216378 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.216359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.223919 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.223895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6vb\" (UniqueName: \"kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb\") pod \"sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.309797 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.309707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:39.452862 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.452835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:15:39.454856 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:15:39.454794 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f54090f_1cf3_453d_baee_d61b2fb03eed.slice/crio-3c921022a5247a18733bcf5a4f58517e2cf7e9ff4f1cb9616d64fda7a6028f74 WatchSource:0}: Error finding container 3c921022a5247a18733bcf5a4f58517e2cf7e9ff4f1cb9616d64fda7a6028f74: Status 404 returned error can't find the container with id 3c921022a5247a18733bcf5a4f58517e2cf7e9ff4f1cb9616d64fda7a6028f74 Jul 16 21:15:39.968803 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.968766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" event={"ID":"2f54090f-1cf3-453d-baee-d61b2fb03eed","Type":"ContainerStarted","Data":"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a"} Jul 16 21:15:39.968803 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.968805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" event={"ID":"2f54090f-1cf3-453d-baee-d61b2fb03eed","Type":"ContainerStarted","Data":"3c921022a5247a18733bcf5a4f58517e2cf7e9ff4f1cb9616d64fda7a6028f74"} Jul 16 21:15:39.990426 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:39.990379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" podStartSLOduration=1.990364686 podStartE2EDuration="1.990364686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:39.988224608 +0000 UTC m=+2538.489305335" watchObservedRunningTime="2026-07-16 21:15:39.990364686 +0000 UTC m=+2538.491445412" Jul 16 21:15:41.114898 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.114859 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" podUID="e80dab05-a740-455b-9826-b8b0a51b1f54" containerName="main" containerID="cri-o://0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10" gracePeriod=45 Jul 16 21:15:41.367917 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.367860 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:15:41.435487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435453 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435490 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435671 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435525 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435671 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435566 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435671 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435671 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435665 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5mzd\" (UniqueName: \"kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd\") pod \"e80dab05-a740-455b-9826-b8b0a51b1f54\" (UID: \"e80dab05-a740-455b-9826-b8b0a51b1f54\") " Jul 16 21:15:41.435858 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435714 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:41.435858 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435763 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache" (OuterVolumeSpecName: "model-cache") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:41.435972 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435941 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home" (OuterVolumeSpecName: "home") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:41.435972 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435951 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.436058 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.435983 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.437845 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.437819 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm" (OuterVolumeSpecName: "dshm") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:15:41.437973 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.437946 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:15:41.438035 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.437995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd" (OuterVolumeSpecName: "kube-api-access-p5mzd") pod "e80dab05-a740-455b-9826-b8b0a51b1f54" (UID: "e80dab05-a740-455b-9826-b8b0a51b1f54"). InnerVolumeSpecName "kube-api-access-p5mzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:15:41.536720 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.536682 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80dab05-a740-455b-9826-b8b0a51b1f54-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.536720 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.536713 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.536720 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.536722 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mzd\" (UniqueName: \"kubernetes.io/projected/e80dab05-a740-455b-9826-b8b0a51b1f54-kube-api-access-p5mzd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.536947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.536730 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80dab05-a740-455b-9826-b8b0a51b1f54-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:15:41.980366 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.980337 2564 generic.go:356] "Generic (PLEG): container finished" podID="e80dab05-a740-455b-9826-b8b0a51b1f54" containerID="0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10" exitCode=0 Jul 16 21:15:41.980578 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.980383 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" Jul 16 21:15:41.980578 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.980430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" event={"ID":"e80dab05-a740-455b-9826-b8b0a51b1f54","Type":"ContainerDied","Data":"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10"} Jul 16 21:15:41.980578 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.980476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv" event={"ID":"e80dab05-a740-455b-9826-b8b0a51b1f54","Type":"ContainerDied","Data":"a709ef85d9326c42ff77a31c58e89ae289e855070733329fbd995fd78f443e92"} Jul 16 21:15:41.980578 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.980499 2564 scope.go:117] "RemoveContainer" containerID="0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10" Jul 16 21:15:41.990334 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.990287 2564 scope.go:117] "RemoveContainer" containerID="0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10" Jul 16 21:15:41.990610 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:15:41.990566 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10\": container with ID starting with 0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10 not found: ID does not exist" containerID="0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10" Jul 16 21:15:41.990676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:41.990626 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10"} err="failed to get container status \"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10\": rpc error: code = NotFound desc = could not find container \"0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10\": container with ID starting with 0c3e0dd2dd9028b40630e6c07a692803a882a9caab661410cd0c26e30c8b3d10 not found: ID does not exist" Jul 16 21:15:42.005238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:42.005200 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:15:42.009173 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:42.009150 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/auth-test-model-09402f-kserve-54c4cf5687-8kxvv"] Jul 16 21:15:42.039468 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:42.039440 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80dab05-a740-455b-9826-b8b0a51b1f54" path="/var/lib/kubelet/pods/e80dab05-a740-455b-9826-b8b0a51b1f54/volumes" Jul 16 21:15:49.310229 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:49.310191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:49.310701 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:49.310288 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:49.322828 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:49.322806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:15:50.039772 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:15:50.039743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:16:02.457041 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:02.457005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:25.304645 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:25.304608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:25.369573 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:25.369536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:52.069952 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:52.069865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:52.171004 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:52.170966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:57.634081 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:57.634042 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:16:57.661935 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:57.661892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:16:58.468958 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:16:58.468918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:17:12.656442 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:12.656386 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" podUID="2f54090f-1cf3-453d-baee-d61b2fb03eed" containerName="main" containerID="cri-o://d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a" gracePeriod=45 Jul 16 21:17:12.912883 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:12.912809 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:17:13.002840 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.002805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.002881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f6vb\" (UniqueName: \"kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.002903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.002929 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.002970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003255 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003051 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home\") pod \"2f54090f-1cf3-453d-baee-d61b2fb03eed\" (UID: \"2f54090f-1cf3-453d-baee-d61b2fb03eed\") " Jul 16 21:17:13.003255 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003145 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:13.003403 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003275 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:13.003463 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home" (OuterVolumeSpecName: "home") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:13.003518 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003467 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.003518 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.003490 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.005149 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.005126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm" (OuterVolumeSpecName: "dshm") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:17:13.005223 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.005154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:17:13.005223 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.005210 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb" (OuterVolumeSpecName: "kube-api-access-6f6vb") pod "2f54090f-1cf3-453d-baee-d61b2fb03eed" (UID: "2f54090f-1cf3-453d-baee-d61b2fb03eed"). InnerVolumeSpecName "kube-api-access-6f6vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:17:13.105013 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.104974 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.105013 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.105008 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f54090f-1cf3-453d-baee-d61b2fb03eed-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.105258 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.105034 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6vb\" (UniqueName: \"kubernetes.io/projected/2f54090f-1cf3-453d-baee-d61b2fb03eed-kube-api-access-6f6vb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.105258 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.105063 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f54090f-1cf3-453d-baee-d61b2fb03eed-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:17:13.465260 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.465221 2564 generic.go:356] "Generic (PLEG): container finished" podID="2f54090f-1cf3-453d-baee-d61b2fb03eed" containerID="d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a" exitCode=0 Jul 16 21:17:13.465469 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.465344 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" Jul 16 21:17:13.465469 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.465340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" event={"ID":"2f54090f-1cf3-453d-baee-d61b2fb03eed","Type":"ContainerDied","Data":"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a"} Jul 16 21:17:13.465590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.465509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9" event={"ID":"2f54090f-1cf3-453d-baee-d61b2fb03eed","Type":"ContainerDied","Data":"3c921022a5247a18733bcf5a4f58517e2cf7e9ff4f1cb9616d64fda7a6028f74"} Jul 16 21:17:13.465590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.465543 2564 scope.go:117] "RemoveContainer" containerID="d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a" Jul 16 21:17:13.476657 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.476629 2564 scope.go:117] "RemoveContainer" containerID="d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a" Jul 16 21:17:13.476978 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:17:13.476949 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a\": container with ID starting with d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a not found: ID does not exist" containerID="d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a" Jul 16 21:17:13.477068 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.476991 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a"} err="failed to get container status \"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a\": rpc error: code = NotFound desc = could not find container \"d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a\": container with ID starting with d80b9c43640b07eb77f21d5ef1f6ddaf4c213535031921f81fe3519cc6eee95a not found: ID does not exist" Jul 16 21:17:13.489285 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.489255 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:17:13.492881 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:13.492852 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/sub-test-model-09402f-kserve-7bb4b8b4d9-xcqw9"] Jul 16 21:17:14.039834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:14.039800 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f54090f-1cf3-453d-baee-d61b2fb03eed" path="/var/lib/kubelet/pods/2f54090f-1cf3-453d-baee-d61b2fb03eed/volumes" Jul 16 21:17:23.625155 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625119 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:17:23.625635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f54090f-1cf3-453d-baee-d61b2fb03eed" containerName="main" Jul 16 21:17:23.625635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625549 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f54090f-1cf3-453d-baee-d61b2fb03eed" containerName="main" Jul 16 21:17:23.625635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625561 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80dab05-a740-455b-9826-b8b0a51b1f54" containerName="main" Jul 16 21:17:23.625635 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625568 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80dab05-a740-455b-9826-b8b0a51b1f54" containerName="main" Jul 16 21:17:23.625767 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625638 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f54090f-1cf3-453d-baee-d61b2fb03eed" containerName="main" Jul 16 21:17:23.625767 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.625648 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80dab05-a740-455b-9826-b8b0a51b1f54" containerName="main" Jul 16 21:17:23.630027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.630004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.636812 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.636786 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"openshift-service-ca.crt\"" Jul 16 21:17:23.637731 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.637715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"rate-test-model-09402f-kserve-self-signed-certs\"" Jul 16 21:17:23.637823 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.637763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"default-dockercfg-pd67w\"" Jul 16 21:17:23.638069 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.638050 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-09402f\"/\"kube-root-ca.crt\"" Jul 16 21:17:23.656207 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.656177 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:17:23.703482 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.703482 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.703712 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnct\" (UniqueName: \"kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.703712 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.703712 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.703712 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.703706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.710888 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.710860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:17:23.804335 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804335 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804593 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804593 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804593 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804593 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnct\" (UniqueName: \"kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804848 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804902 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.804947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.804920 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.806831 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.806809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.807050 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.807033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.814698 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.814676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnct\" (UniqueName: \"kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct\") pod \"rate-test-model-09402f-kserve-86987cccc9-7xvv8\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:23.940019 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:23.939921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:24.093830 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:17:24.093790 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb608818e_f57c_4fa6_a676_ba1bcdb1f27c.slice/crio-47aa7abf58e5c8aac853f64b7488586afa323a4e1178792e78b00b83574f1f2b WatchSource:0}: Error finding container 47aa7abf58e5c8aac853f64b7488586afa323a4e1178792e78b00b83574f1f2b: Status 404 returned error can't find the container with id 47aa7abf58e5c8aac853f64b7488586afa323a4e1178792e78b00b83574f1f2b Jul 16 21:17:24.093981 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:24.093836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:17:24.541610 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:24.541575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" event={"ID":"b608818e-f57c-4fa6-a676-ba1bcdb1f27c","Type":"ContainerStarted","Data":"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32"} Jul 16 21:17:24.541610 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:24.541616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" event={"ID":"b608818e-f57c-4fa6-a676-ba1bcdb1f27c","Type":"ContainerStarted","Data":"47aa7abf58e5c8aac853f64b7488586afa323a4e1178792e78b00b83574f1f2b"} Jul 16 21:17:24.562697 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:24.562640 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" podStartSLOduration=1.562623883 podStartE2EDuration="1.562623883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:17:24.561288875 +0000 UTC m=+2643.062369627" watchObservedRunningTime="2026-07-16 21:17:24.562623883 +0000 UTC m=+2643.063704609" Jul 16 21:17:33.940734 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:33.940690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:33.941238 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:33.940759 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:33.953243 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:33.953207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:34.607409 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:34.607382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:17:52.100084 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.100050 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:17:52.103709 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.103683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.106400 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.106365 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"default-dockercfg-9v6zv\"" Jul 16 21:17:52.106572 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.106474 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"openshift-service-ca.crt\"" Jul 16 21:17:52.107708 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.107688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"rate-test-model-69d8e7-kserve-self-signed-certs\"" Jul 16 21:17:52.107821 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.107736 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-69d8e7\"/\"kube-root-ca.crt\"" Jul 16 21:17:52.112757 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.112726 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:17:52.134753 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.134722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:17:52.159947 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.159913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.160125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.159973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.160125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.160035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.160125 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.160065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.160234 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.160114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.160234 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.160191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcj7p\" (UniqueName: \"kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261401 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261598 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261598 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261598 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261598 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.261807 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcj7p\" (UniqueName: \"kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.262037 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261916 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.262037 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.261974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.262218 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.262190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.264659 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.264635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.264870 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.264852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.271461 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.271433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcj7p\" (UniqueName: \"kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p\") pod \"rate-test-model-69d8e7-kserve-578fc8467-ctjcb\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.415407 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.415280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:17:52.564996 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.564968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:17:52.567045 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:17:52.567019 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ed0691_aa04_488f_9fd9_701c77213e6e.slice/crio-147e4e2ea1e3c7e40bac6e48daeb9addb554fbc90ebb8fbd8301477ffb508e90 WatchSource:0}: Error finding container 147e4e2ea1e3c7e40bac6e48daeb9addb554fbc90ebb8fbd8301477ffb508e90: Status 404 returned error can't find the container with id 147e4e2ea1e3c7e40bac6e48daeb9addb554fbc90ebb8fbd8301477ffb508e90 Jul 16 21:17:52.694713 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.694620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" event={"ID":"f0ed0691-aa04-488f-9fd9-701c77213e6e","Type":"ContainerStarted","Data":"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf"} Jul 16 21:17:52.694713 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.694671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" event={"ID":"f0ed0691-aa04-488f-9fd9-701c77213e6e","Type":"ContainerStarted","Data":"147e4e2ea1e3c7e40bac6e48daeb9addb554fbc90ebb8fbd8301477ffb508e90"} Jul 16 21:17:52.716848 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:17:52.716793 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" podStartSLOduration=0.71677643 podStartE2EDuration="716.77643ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:17:52.715134669 +0000 UTC m=+2671.216215395" watchObservedRunningTime="2026-07-16 21:17:52.71677643 +0000 UTC m=+2671.217857156" Jul 16 21:18:02.416450 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:02.416409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:02.416966 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:02.416565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:02.429600 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:02.429566 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:02.757608 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:02.757579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:13.863700 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:13.863623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:13.963400 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:13.963365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:21.742166 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:21.742125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:21.848095 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:21.848062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:22.142825 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:22.142739 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:18:22.154154 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:22.154131 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j6xlh_2cb64f02-7b87-4004-9126-347a2a912e2f/ovn-acl-logging/0.log" Jul 16 21:18:41.390641 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:41.390609 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:18:41.418032 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:41.418004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:42.447978 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:42.447946 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:18:42.488589 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:42.488558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:18:56.413545 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.413498 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" podUID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" containerName="main" containerID="cri-o://3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32" gracePeriod=45 Jul 16 21:18:56.678930 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.678863 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:18:56.776655 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776619 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.776860 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776707 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dnct\" (UniqueName: \"kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.776860 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.776860 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.777044 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.777044 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776961 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache\") pod \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\" (UID: \"b608818e-f57c-4fa6-a676-ba1bcdb1f27c\") " Jul 16 21:18:56.777044 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.776985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:56.777044 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.777029 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home" (OuterVolumeSpecName: "home") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:56.777350 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.777324 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:56.777350 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.777346 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:56.777350 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.777333 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:56.779172 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.779148 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct" (OuterVolumeSpecName: "kube-api-access-5dnct") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "kube-api-access-5dnct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:18:56.779286 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.779188 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm" (OuterVolumeSpecName: "dshm") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:56.779286 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.779189 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b608818e-f57c-4fa6-a676-ba1bcdb1f27c" (UID: "b608818e-f57c-4fa6-a676-ba1bcdb1f27c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:18:56.878672 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.878628 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:56.878672 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.878669 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:56.878885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.878684 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnct\" (UniqueName: \"kubernetes.io/projected/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-kube-api-access-5dnct\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:56.878885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:56.878695 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b608818e-f57c-4fa6-a676-ba1bcdb1f27c-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.032050 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.032015 2564 generic.go:356] "Generic (PLEG): container finished" podID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" containerID="3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32" exitCode=0 Jul 16 21:18:57.032221 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.032076 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" Jul 16 21:18:57.032221 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.032072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" event={"ID":"b608818e-f57c-4fa6-a676-ba1bcdb1f27c","Type":"ContainerDied","Data":"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32"} Jul 16 21:18:57.032221 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.032196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8" event={"ID":"b608818e-f57c-4fa6-a676-ba1bcdb1f27c","Type":"ContainerDied","Data":"47aa7abf58e5c8aac853f64b7488586afa323a4e1178792e78b00b83574f1f2b"} Jul 16 21:18:57.032221 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.032221 2564 scope.go:117] "RemoveContainer" containerID="3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32" Jul 16 21:18:57.041825 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.041805 2564 scope.go:117] "RemoveContainer" containerID="3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32" Jul 16 21:18:57.042112 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:18:57.042077 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32\": container with ID starting with 3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32 not found: ID does not exist" containerID="3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32" Jul 16 21:18:57.042156 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.042124 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32"} err="failed to get container status \"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32\": rpc error: code = NotFound desc = could not find container \"3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32\": container with ID starting with 3429baf171e19b61f7b42d8698725997610286bf4006cd0db21bad135fcdba32 not found: ID does not exist" Jul 16 21:18:57.055288 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.055246 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:18:57.060206 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.060178 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-09402f/rate-test-model-09402f-kserve-86987cccc9-7xvv8"] Jul 16 21:18:57.468371 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.468246 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" podUID="f0ed0691-aa04-488f-9fd9-701c77213e6e" containerName="main" containerID="cri-o://b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf" gracePeriod=45 Jul 16 21:18:57.734034 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.733966 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:57.889558 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889511 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcj7p\" (UniqueName: \"kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889677 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889740 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs\") pod \"f0ed0691-aa04-488f-9fd9-701c77213e6e\" (UID: \"f0ed0691-aa04-488f-9fd9-701c77213e6e\") " Jul 16 21:18:57.889950 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:57.889950 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889858 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home" (OuterVolumeSpecName: "home") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:57.890021 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.889995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:57.890076 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.890060 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.890135 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.890083 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.890135 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.890097 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.891988 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.891956 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p" (OuterVolumeSpecName: "kube-api-access-xcj7p") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "kube-api-access-xcj7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:18:57.891988 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.891977 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:18:57.892136 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.892048 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm" (OuterVolumeSpecName: "dshm") pod "f0ed0691-aa04-488f-9fd9-701c77213e6e" (UID: "f0ed0691-aa04-488f-9fd9-701c77213e6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:18:57.990828 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.990725 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcj7p\" (UniqueName: \"kubernetes.io/projected/f0ed0691-aa04-488f-9fd9-701c77213e6e-kube-api-access-xcj7p\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.990828 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.990771 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0ed0691-aa04-488f-9fd9-701c77213e6e-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:57.990828 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:57.990784 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0ed0691-aa04-488f-9fd9-701c77213e6e-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:18:58.039978 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.039944 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0ed0691-aa04-488f-9fd9-701c77213e6e" containerID="b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf" exitCode=0 Jul 16 21:18:58.040161 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.040049 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" Jul 16 21:18:58.040837 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.040810 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" path="/var/lib/kubelet/pods/b608818e-f57c-4fa6-a676-ba1bcdb1f27c/volumes" Jul 16 21:18:58.041284 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.041262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" event={"ID":"f0ed0691-aa04-488f-9fd9-701c77213e6e","Type":"ContainerDied","Data":"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf"} Jul 16 21:18:58.041407 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.041329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb" event={"ID":"f0ed0691-aa04-488f-9fd9-701c77213e6e","Type":"ContainerDied","Data":"147e4e2ea1e3c7e40bac6e48daeb9addb554fbc90ebb8fbd8301477ffb508e90"} Jul 16 21:18:58.041407 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.041361 2564 scope.go:117] "RemoveContainer" containerID="b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf" Jul 16 21:18:58.051198 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.051141 2564 scope.go:117] "RemoveContainer" containerID="b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf" Jul 16 21:18:58.051536 ip-10-0-131-179 kubenswrapper[2564]: E0716 21:18:58.051506 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf\": container with ID starting with b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf not found: ID does not exist" containerID="b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf" Jul 16 21:18:58.051633 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.051545 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf"} err="failed to get container status \"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf\": rpc error: code = NotFound desc = could not find container \"b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf\": container with ID starting with b6990f8950c1078ffba3cc7263a795e7b02158ed2344ab0454ad941211f53fdf not found: ID does not exist" Jul 16 21:18:58.070924 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.070894 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:18:58.078228 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.078202 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-69d8e7/rate-test-model-69d8e7-kserve-578fc8467-ctjcb"] Jul 16 21:18:58.227503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227468 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7154-140581941127040"] Jul 16 21:18:58.227909 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227894 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" containerName="main" Jul 16 21:18:58.227954 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227915 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" containerName="main" Jul 16 21:18:58.227954 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227926 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ed0691-aa04-488f-9fd9-701c77213e6e" containerName="main" Jul 16 21:18:58.227954 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227932 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ed0691-aa04-488f-9fd9-701c77213e6e" containerName="main" Jul 16 21:18:58.228049 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.227997 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b608818e-f57c-4fa6-a676-ba1bcdb1f27c" containerName="main" Jul 16 21:18:58.228049 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.228010 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ed0691-aa04-488f-9fd9-701c77213e6e" containerName="main" Jul 16 21:18:58.230964 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.230947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:18:58.234643 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.234623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fq898\"" Jul 16 21:18:58.241186 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.241128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"] Jul 16 21:18:58.292809 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.292768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5kps\" (UniqueName: \"kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps\") pod \"test-curl-7154-140581941127040\" (UID: \"81d82fb3-c204-48c5-8895-27b7805df411\") " pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:18:58.394437 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.394396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5kps\" (UniqueName: \"kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps\") pod \"test-curl-7154-140581941127040\" (UID: \"81d82fb3-c204-48c5-8895-27b7805df411\") " pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:18:58.404770 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.404740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5kps\" (UniqueName: \"kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps\") pod \"test-curl-7154-140581941127040\" (UID: \"81d82fb3-c204-48c5-8895-27b7805df411\") " pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:18:58.540400 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.540278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:18:58.681465 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.681437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"] Jul 16 21:18:58.683402 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:18:58.683372 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81d82fb3_c204_48c5_8895_27b7805df411.slice/crio-b954876f86f0c8867b43206dfa7ff1c93afcfe087fa0e4a17d61b04524efff45 WatchSource:0}: Error finding container b954876f86f0c8867b43206dfa7ff1c93afcfe087fa0e4a17d61b04524efff45: Status 404 returned error can't find the container with id b954876f86f0c8867b43206dfa7ff1c93afcfe087fa0e4a17d61b04524efff45 Jul 16 21:18:58.684683 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:58.684665 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:18:59.049733 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:18:59.049660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941127040" event={"ID":"81d82fb3-c204-48c5-8895-27b7805df411","Type":"ContainerStarted","Data":"b954876f86f0c8867b43206dfa7ff1c93afcfe087fa0e4a17d61b04524efff45"} Jul 16 21:19:00.043580 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:00.043542 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ed0691-aa04-488f-9fd9-701c77213e6e" path="/var/lib/kubelet/pods/f0ed0691-aa04-488f-9fd9-701c77213e6e/volumes" Jul 16 21:19:00.056951 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:00.056915 2564 generic.go:356] "Generic (PLEG): container finished" podID="81d82fb3-c204-48c5-8895-27b7805df411" containerID="3d4bf36478515e56b47d85fd63cd8e33a3faae2cc9ec08350e2fe6a4488c2ece" exitCode=0 Jul 16 21:19:00.057111 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:00.057007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7154-140581941127040" event={"ID":"81d82fb3-c204-48c5-8895-27b7805df411","Type":"ContainerDied","Data":"3d4bf36478515e56b47d85fd63cd8e33a3faae2cc9ec08350e2fe6a4488c2ece"} Jul 16 21:19:01.206463 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.206433 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:19:01.230277 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.230250 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7154-140581941127040_81d82fb3-c204-48c5-8895-27b7805df411/test-curl-7154-140581941127040/0.log" Jul 16 21:19:01.257763 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.257732 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"] Jul 16 21:19:01.262182 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.262151 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7154-140581941127040"] Jul 16 21:19:01.320570 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.320483 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5kps\" (UniqueName: \"kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps\") pod \"81d82fb3-c204-48c5-8895-27b7805df411\" (UID: \"81d82fb3-c204-48c5-8895-27b7805df411\") " Jul 16 21:19:01.322760 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.322737 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps" (OuterVolumeSpecName: "kube-api-access-b5kps") pod "81d82fb3-c204-48c5-8895-27b7805df411" (UID: "81d82fb3-c204-48c5-8895-27b7805df411"). InnerVolumeSpecName "kube-api-access-b5kps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:19:01.422032 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.422000 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5kps\" (UniqueName: \"kubernetes.io/projected/81d82fb3-c204-48c5-8895-27b7805df411-kube-api-access-b5kps\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:19:01.458572 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.458534 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf"] Jul 16 21:19:01.458968 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.458955 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d82fb3-c204-48c5-8895-27b7805df411" containerName="test-curl-7154-140581941127040" Jul 16 21:19:01.459019 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.458970 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d82fb3-c204-48c5-8895-27b7805df411" containerName="test-curl-7154-140581941127040" Jul 16 21:19:01.459058 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.459034 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d82fb3-c204-48c5-8895-27b7805df411" containerName="test-curl-7154-140581941127040" Jul 16 21:19:01.462074 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.462053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:01.465000 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.464983 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:19:01.465235 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.465222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fcvh8\"" Jul 16 21:19:01.471731 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.471707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf"] Jul 16 21:19:01.623881 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.623780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf\" (UID: \"9be8e99e-4152-495b-8efe-268e36e22148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:01.725128 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.725083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf\" (UID: \"9be8e99e-4152-495b-8efe-268e36e22148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:01.725779 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.725758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf\" (UID: \"9be8e99e-4152-495b-8efe-268e36e22148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:01.773134 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.773096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:01.912403 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:01.912379 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf"] Jul 16 21:19:01.915052 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:19:01.915020 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be8e99e_4152_495b_8efe_268e36e22148.slice/crio-6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1 WatchSource:0}: Error finding container 6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1: Status 404 returned error can't find the container with id 6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1 Jul 16 21:19:02.041145 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:02.041110 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d82fb3-c204-48c5-8895-27b7805df411" path="/var/lib/kubelet/pods/81d82fb3-c204-48c5-8895-27b7805df411/volumes" Jul 16 21:19:02.071542 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:02.071511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" event={"ID":"9be8e99e-4152-495b-8efe-268e36e22148","Type":"ContainerStarted","Data":"6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1"} Jul 16 21:19:02.072915 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:02.072891 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7154-140581941127040" Jul 16 21:19:02.073036 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:02.072944 2564 scope.go:117] "RemoveContainer" containerID="3d4bf36478515e56b47d85fd63cd8e33a3faae2cc9ec08350e2fe6a4488c2ece" Jul 16 21:19:03.079636 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:03.079595 2564 generic.go:356] "Generic (PLEG): container finished" podID="9be8e99e-4152-495b-8efe-268e36e22148" containerID="02fc46b95a9ae0da924aa19c42c4c9008076dbf56fcfaae0c81467290b510bcb" exitCode=0 Jul 16 21:19:03.080050 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:03.079682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" event={"ID":"9be8e99e-4152-495b-8efe-268e36e22148","Type":"ContainerDied","Data":"02fc46b95a9ae0da924aa19c42c4c9008076dbf56fcfaae0c81467290b510bcb"} Jul 16 21:19:04.222088 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:04.222060 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:04.351214 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:04.351123 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca\") pod \"9be8e99e-4152-495b-8efe-268e36e22148\" (UID: \"9be8e99e-4152-495b-8efe-268e36e22148\") " Jul 16 21:19:04.351564 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:04.351534 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9be8e99e-4152-495b-8efe-268e36e22148" (UID: "9be8e99e-4152-495b-8efe-268e36e22148"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:19:04.451988 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:04.451955 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be8e99e-4152-495b-8efe-268e36e22148-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:19:05.092465 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:05.092432 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" Jul 16 21:19:05.092465 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:05.092459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf" event={"ID":"9be8e99e-4152-495b-8efe-268e36e22148","Type":"ContainerDied","Data":"6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1"} Jul 16 21:19:05.092677 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:05.092506 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e041ba27daec0678bcfb7a24f2e65eb4ff208e993e0905af9f899fc640cd4f1" Jul 16 21:19:11.571195 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:11.571157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:16.677260 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:16.677227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:17.105596 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:17.105566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:23.567025 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:23.566986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:23.673270 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:23.673237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:23.982057 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:23.982022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:24.205889 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.205857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:24.269095 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.269011 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg"] Jul 16 21:19:24.269438 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.269425 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be8e99e-4152-495b-8efe-268e36e22148" containerName="revoke-keys" Jul 16 21:19:24.269487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.269439 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be8e99e-4152-495b-8efe-268e36e22148" containerName="revoke-keys" Jul 16 21:19:24.269523 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.269501 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be8e99e-4152-495b-8efe-268e36e22148" containerName="revoke-keys" Jul 16 21:19:24.274132 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.274103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:24.278409 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.278375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fcvh8\"" Jul 16 21:19:24.278736 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.278714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 21:19:24.302847 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.302801 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg"] Jul 16 21:19:24.347011 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.346975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg\" (UID: \"c4ca9225-3ee5-4492-8d57-bb50516b261d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:24.448417 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.448376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg\" (UID: \"c4ca9225-3ee5-4492-8d57-bb50516b261d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:24.449067 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.449048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg\" (UID: \"c4ca9225-3ee5-4492-8d57-bb50516b261d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:24.585284 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.585197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:24.933019 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:24.932943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg"] Jul 16 21:19:24.941786 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:19:24.941758 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ca9225_3ee5_4492_8d57_bb50516b261d.slice/crio-90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a WatchSource:0}: Error finding container 90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a: Status 404 returned error can't find the container with id 90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a Jul 16 21:19:25.196209 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:25.196116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" event={"ID":"c4ca9225-3ee5-4492-8d57-bb50516b261d","Type":"ContainerStarted","Data":"90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a"} Jul 16 21:19:26.203565 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:26.203533 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4ca9225-3ee5-4492-8d57-bb50516b261d" containerID="d98ffcd050fbaa396dd683dbf1598d46a034f32aca7c549aba3423589025a75b" exitCode=0 Jul 16 21:19:26.203972 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:26.203589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" event={"ID":"c4ca9225-3ee5-4492-8d57-bb50516b261d","Type":"ContainerDied","Data":"d98ffcd050fbaa396dd683dbf1598d46a034f32aca7c549aba3423589025a75b"} Jul 16 21:19:27.348113 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:27.348090 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:27.478463 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:27.478433 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca\") pod \"c4ca9225-3ee5-4492-8d57-bb50516b261d\" (UID: \"c4ca9225-3ee5-4492-8d57-bb50516b261d\") " Jul 16 21:19:27.478785 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:27.478762 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4ca9225-3ee5-4492-8d57-bb50516b261d" (UID: "c4ca9225-3ee5-4492-8d57-bb50516b261d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:19:27.579928 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:27.579886 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ca9225-3ee5-4492-8d57-bb50516b261d-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Jul 16 21:19:28.214384 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:28.214350 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" Jul 16 21:19:28.214384 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:28.214371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg" event={"ID":"c4ca9225-3ee5-4492-8d57-bb50516b261d","Type":"ContainerDied","Data":"90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a"} Jul 16 21:19:28.214595 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:28.214425 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90508f9b660122670331fffbe7c0f84cbb890658ca0b4a9265ef8fa52bb0f84a" Jul 16 21:19:34.324518 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:34.324473 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk"] Jul 16 21:19:34.327898 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:34.327863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:34.328051 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:34.328012 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-69d8e7-29737275-pd7nk"] Jul 16 21:19:36.039070 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:36.039030 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caab534a-3f82-455b-bb9f-255a3de57210" path="/var/lib/kubelet/pods/caab534a-3f82-455b-bb9f-255a3de57210/volumes" Jul 16 21:19:39.439119 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:39.439086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:39.827133 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:39.827099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:45.961066 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:45.960977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:46.062026 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:46.061998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:46.237345 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:46.237315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:46.478392 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:46.478354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:19:46.913914 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:46.913886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d66667464-lmjqj_6e3f6b26-df01-410e-9f39-da359c8471ad/authorino/0.log" Jul 16 21:19:52.453551 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:52.453521 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-krhtj_c684a01c-63a0-4c84-b5f6-56dd8a199320/manager/0.log" Jul 16 21:19:52.803754 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:52.803729 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-j2k9j_c8eb74e0-600a-4af0-96b3-47143c987d14/server/0.log" Jul 16 21:19:52.923178 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:52.923149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-x5l4z_ca05d45f-a8fb-452d-b979-f2905a1eb08e/manager/0.log" Jul 16 21:19:53.173506 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:53.173433 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-bw6kf_af6d3e47-493e-4830-8a86-8d173191a8e4/copy-manifests/0.log" Jul 16 21:19:53.188699 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:53.188667 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-bw6kf_af6d3e47-493e-4830-8a86-8d173191a8e4/manager/0.log" Jul 16 21:19:54.053503 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:54.053464 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737275-877v2_825b70c1-3874-4b46-b2bf-ec0a71e1ca50/cleanup/0.log" Jul 16 21:19:54.513041 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:54.513011 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-a896c7-9558m_f6de7096-74d6-4c10-b5cc-fe5a233f81e2/revoke-keys/0.log" Jul 16 21:19:55.313457 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:55.313429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-09402f-2f8qf_9be8e99e-4152-495b-8efe-268e36e22148/revoke-keys/0.log" Jul 16 21:19:55.430825 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:55.430794 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-69d8e7-94clg_c4ca9225-3ee5-4492-8d57-bb50516b261d/revoke-keys/0.log" Jul 16 21:19:56.864970 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:56.864940 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d66667464-lmjqj_6e3f6b26-df01-410e-9f39-da359c8471ad/authorino/0.log" Jul 16 21:19:56.989547 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:56.989515 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zhzx8_9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e/manager/0.log" Jul 16 21:19:57.103973 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:57.103949 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-twhkf_e4dea57f-2641-4933-97d5-c86f67aaef7b/manager/0.log" Jul 16 21:19:57.332623 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:57.332595 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t9558_7dbd4fa1-91ab-436d-8dff-2bbb32aa171f/registry-server/0.log" Jul 16 21:19:57.461511 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:57.461480 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qrdc2_48aeacaa-0e70-4a85-91ba-7a59aebc381b/manager/0.log" Jul 16 21:19:57.697825 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:57.697739 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pb8kq_fe0eba06-dc02-4011-832f-b0108fd2458a/manager/0.log" Jul 16 21:19:58.081590 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:58.081564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45cpmr_a80aa256-e1ca-4412-8aa3-780d26014fff/istio-proxy/0.log" Jul 16 21:19:58.432146 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:58.432064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-vwjwn_a8675372-fc09-480b-8fad-dead11a91eca/kube-auth-proxy/0.log" Jul 16 21:19:58.549473 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:58.549441 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/1.log" Jul 16 21:19:58.791175 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:58.791142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hjtkn_3666a563-e014-4fbe-ad65-ef0a378cfa88/payload-processing/0.log" Jul 16 21:19:59.371209 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:59.371181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2kvwc_5e09ce08-a4d0-48b1-8dd9-1d3ebb3b7461/main/0.log" Jul 16 21:19:59.482739 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:59.482709 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-988cr_a06addce-875a-46e5-b2fb-2fd58f5c828b/main/0.log" Jul 16 21:19:59.822524 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:19:59.822497 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tm5rs_c3dfe486-9ebb-4522-87f9-025c3de18ad9/main/0.log" Jul 16 21:20:05.222686 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:05.222647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:20:05.291611 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:05.291569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:20:08.257204 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:08.257165 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vz888_157e505d-0be0-4b00-88aa-2d9ab9a68b15/global-pull-secret-syncer/0.log" Jul 16 21:20:08.332487 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:08.332458 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" Jul 16 21:20:08.442321 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:08.442271 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-179.ec2.internal_2d0a72e3dbfc85402677ef718ad352e5/haproxy/0.log" Jul 16 21:20:14.164676 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.164631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d66667464-lmjqj_6e3f6b26-df01-410e-9f39-da359c8471ad/authorino/0.log" Jul 16 21:20:14.198501 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.198462 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zhzx8_9b4c3b83-9b6e-44cc-b9f2-16c3475dcf2e/manager/0.log" Jul 16 21:20:14.232641 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.232607 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-twhkf_e4dea57f-2641-4933-97d5-c86f67aaef7b/manager/0.log" Jul 16 21:20:14.317531 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.317481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t9558_7dbd4fa1-91ab-436d-8dff-2bbb32aa171f/registry-server/0.log" Jul 16 21:20:14.561699 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.561652 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qrdc2_48aeacaa-0e70-4a85-91ba-7a59aebc381b/manager/0.log" Jul 16 21:20:14.651626 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:14.651593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pb8kq_fe0eba06-dc02-4011-832f-b0108fd2458a/manager/0.log" Jul 16 21:20:16.269924 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.269891 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/alertmanager/0.log" Jul 16 21:20:16.331031 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.331000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/config-reloader/0.log" Jul 16 21:20:16.361534 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.361506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/kube-rbac-proxy-web/0.log" Jul 16 21:20:16.398030 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.397998 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/kube-rbac-proxy/0.log" Jul 16 21:20:16.430584 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.430556 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/kube-rbac-proxy-metric/0.log" Jul 16 21:20:16.457024 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.456994 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/prom-label-proxy/0.log" Jul 16 21:20:16.486577 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.486550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7aa42b13-9d36-432b-8cc7-e8f7219114bb/init-config-reloader/0.log" Jul 16 21:20:16.778279 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.778204 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gckch_d308c4d9-cd87-4ee2-9e04-890e7afb2f3b/node-exporter/0.log" Jul 16 21:20:16.800999 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.800972 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gckch_d308c4d9-cd87-4ee2-9e04-890e7afb2f3b/kube-rbac-proxy/0.log" Jul 16 21:20:16.824073 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.824048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gckch_d308c4d9-cd87-4ee2-9e04-890e7afb2f3b/init-textfile/0.log" Jul 16 21:20:16.933126 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.933074 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tc7r5_100cea57-ab54-439a-a514-492e7fff2f42/kube-rbac-proxy-main/0.log" Jul 16 21:20:16.956047 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.956021 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tc7r5_100cea57-ab54-439a-a514-492e7fff2f42/kube-rbac-proxy-self/0.log" Jul 16 21:20:16.980673 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:16.980646 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tc7r5_100cea57-ab54-439a-a514-492e7fff2f42/openshift-state-metrics/0.log" Jul 16 21:20:17.301201 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:17.301174 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84d55bcd7c-4gwtx_ca53ca11-351a-4add-b2b7-4d17c092d0fa/telemeter-client/0.log" Jul 16 21:20:17.332744 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:17.332710 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84d55bcd7c-4gwtx_ca53ca11-351a-4add-b2b7-4d17c092d0fa/reload/0.log" Jul 16 21:20:17.361986 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:17.361960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84d55bcd7c-4gwtx_ca53ca11-351a-4add-b2b7-4d17c092d0fa/kube-rbac-proxy/0.log" Jul 16 21:20:18.298200 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.298164 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9"] Jul 16 21:20:18.298573 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.298560 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ca9225-3ee5-4492-8d57-bb50516b261d" containerName="revoke-keys" Jul 16 21:20:18.298627 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.298575 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ca9225-3ee5-4492-8d57-bb50516b261d" containerName="revoke-keys" Jul 16 21:20:18.298662 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.298655 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ca9225-3ee5-4492-8d57-bb50516b261d" containerName="revoke-keys" Jul 16 21:20:18.301706 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.301689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.304795 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.304766 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmclr\"/\"openshift-service-ca.crt\"" Jul 16 21:20:18.304962 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.304801 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmclr\"/\"kube-root-ca.crt\"" Jul 16 21:20:18.305826 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.305809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rmclr\"/\"default-dockercfg-g7srr\"" Jul 16 21:20:18.313910 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.313882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9"] Jul 16 21:20:18.443459 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.443418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8q2v\" (UniqueName: \"kubernetes.io/projected/4e6a961d-0479-4d26-8be6-3d198d33c93d-kube-api-access-v8q2v\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.443459 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.443466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-podres\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.443674 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.443574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-lib-modules\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.443674 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.443640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-sys\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.443749 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.443704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-proc\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544574 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-sys\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-proc\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8q2v\" (UniqueName: \"kubernetes.io/projected/4e6a961d-0479-4d26-8be6-3d198d33c93d-kube-api-access-v8q2v\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-podres\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-proc\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-sys\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.544783 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-lib-modules\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.545075 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-podres\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.545075 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.544891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e6a961d-0479-4d26-8be6-3d198d33c93d-lib-modules\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.553222 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.553154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8q2v\" (UniqueName: \"kubernetes.io/projected/4e6a961d-0479-4d26-8be6-3d198d33c93d-kube-api-access-v8q2v\") pod \"perf-node-gather-daemonset-mxpt9\" (UID: \"4e6a961d-0479-4d26-8be6-3d198d33c93d\") " pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.612240 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.612204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:18.752688 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:18.752660 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9"] Jul 16 21:20:18.755142 ip-10-0-131-179 kubenswrapper[2564]: W0716 21:20:18.755110 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e6a961d_0479_4d26_8be6_3d198d33c93d.slice/crio-8bd0d5019daab88c960af032cc9a857e6201d865d85ef5f2642981838dfc538d WatchSource:0}: Error finding container 8bd0d5019daab88c960af032cc9a857e6201d865d85ef5f2642981838dfc538d: Status 404 returned error can't find the container with id 8bd0d5019daab88c960af032cc9a857e6201d865d85ef5f2642981838dfc538d Jul 16 21:20:19.488742 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:19.488708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" event={"ID":"4e6a961d-0479-4d26-8be6-3d198d33c93d","Type":"ContainerStarted","Data":"74082081e92acb5eec614149e8f9302fbe08c4107c466a6eb41a3fca144d467e"} Jul 16 21:20:19.489164 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:19.488760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" event={"ID":"4e6a961d-0479-4d26-8be6-3d198d33c93d","Type":"ContainerStarted","Data":"8bd0d5019daab88c960af032cc9a857e6201d865d85ef5f2642981838dfc538d"} Jul 16 21:20:19.489164 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:19.488864 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:19.506572 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:19.506522 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" podStartSLOduration=1.5065059889999999 podStartE2EDuration="1.506505989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:20:19.504566629 +0000 UTC m=+2818.005647351" watchObservedRunningTime="2026-07-16 21:20:19.506505989 +0000 UTC m=+2818.007586716" Jul 16 21:20:21.223834 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:21.223802 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjhm4_86d018bd-ec4f-4ac6-9dd7-34a0c2087f81/dns/0.log" Jul 16 21:20:21.245846 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:21.245817 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjhm4_86d018bd-ec4f-4ac6-9dd7-34a0c2087f81/kube-rbac-proxy/0.log" Jul 16 21:20:21.373271 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:21.373238 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f7sgr_ddf4c0ed-54c3-4ad0-b57f-57b227239cdb/dns-node-resolver/0.log" Jul 16 21:20:21.699212 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:21.699130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:20:21.965163 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:21.965079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-657968875d-v8h6h_c5502967-22ad-415d-9ca3-d4a20e5893e8/registry/0.log" Jul 16 21:20:22.042980 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:22.042956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfs5x_57063013-98eb-4fb8-a531-c9053a008579/node-ca/0.log" Jul 16 21:20:22.631946 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:22.631913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrzzd_e6d1141b-91ef-4a48-94a9-d33108ccecee/konnectivity-agent/0.log" limit=1024 Jul 16 21:20:22.981027 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:22.980982 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45cpmr_a80aa256-e1ca-4412-8aa3-780d26014fff/istio-proxy/0.log" Jul 16 21:20:23.179857 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:23.179820 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-vwjwn_a8675372-fc09-480b-8fad-dead11a91eca/kube-auth-proxy/0.log" Jul 16 21:20:23.207042 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:23.207013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/0.log" Jul 16 21:20:23.240158 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:23.240070 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/1.log" Jul 16 21:20:23.309495 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:23.309458 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hjtkn_3666a563-e014-4fbe-ad65-ef0a378cfa88/payload-processing/0.log" Jul 16 21:20:24.003630 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:24.003602 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2xs5q_37db4d2c-a03f-43be-bac1-a659cb4211b0/serve-healthcheck-canary/0.log" Jul 16 21:20:24.637551 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:24.637520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rfz7_71869283-694c-4d79-8fc0-e38a3e65c3e7/kube-rbac-proxy/0.log" Jul 16 21:20:24.660404 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:24.660377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rfz7_71869283-694c-4d79-8fc0-e38a3e65c3e7/exporter/0.log" Jul 16 21:20:24.686442 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:24.686413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rfz7_71869283-694c-4d79-8fc0-e38a3e65c3e7/extractor/0.log" Jul 16 21:20:25.504391 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:25.504357 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rmclr/perf-node-gather-daemonset-mxpt9" Jul 16 21:20:27.008847 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:27.008821 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-krhtj_c684a01c-63a0-4c84-b5f6-56dd8a199320/manager/0.log" Jul 16 21:20:27.329130 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:27.329099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-j2k9j_c8eb74e0-600a-4af0-96b3-47143c987d14/server/0.log" Jul 16 21:20:27.406016 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:27.405980 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-x5l4z_ca05d45f-a8fb-452d-b979-f2905a1eb08e/manager/0.log" Jul 16 21:20:27.537688 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:27.537653 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-bw6kf_af6d3e47-493e-4830-8a86-8d173191a8e4/manager/0.log" Jul 16 21:20:27.556483 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:27.556455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6477549b-bw6kf_af6d3e47-493e-4830-8a86-8d173191a8e4/copy-manifests/0.log" Jul 16 21:20:28.772364 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:28.772329 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45cpmr_a80aa256-e1ca-4412-8aa3-780d26014fff/istio-proxy/0.log" Jul 16 21:20:28.957241 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:28.957208 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-vwjwn_a8675372-fc09-480b-8fad-dead11a91eca/kube-auth-proxy/0.log" Jul 16 21:20:28.989205 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:28.989178 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/0.log" Jul 16 21:20:29.018162 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:29.018117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/1.log" Jul 16 21:20:29.097099 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:29.097019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hjtkn_3666a563-e014-4fbe-ad65-ef0a378cfa88/payload-processing/0.log" Jul 16 21:20:29.120295 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:29.120261 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gkxzf_3ed5ffd9-b170-495f-b06f-67464b9c92af/openshift-lws-operator/0.log" Jul 16 21:20:30.398963 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:30.398881 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45cpmr_a80aa256-e1ca-4412-8aa3-780d26014fff/istio-proxy/0.log" Jul 16 21:20:30.595023 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:30.594993 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-559f5f697c-vwjwn_a8675372-fc09-480b-8fad-dead11a91eca/kube-auth-proxy/0.log" Jul 16 21:20:30.619996 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:30.619966 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/0.log" Jul 16 21:20:30.651667 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:30.651580 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xfnzl_5a616930-461e-40da-8b27-e6052c45ccc0/istio-proxy/1.log" Jul 16 21:20:30.695376 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:30.695346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hjtkn_3666a563-e014-4fbe-ad65-ef0a378cfa88/payload-processing/0.log" Jul 16 21:20:35.704353 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.704293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4zjhb_ee1d15ed-9d78-450f-9a27-ebd324a0394c/kube-multus/0.log" Jul 16 21:20:35.768549 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.768516 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/kube-multus-additional-cni-plugins/0.log" Jul 16 21:20:35.793799 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.793772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/egress-router-binary-copy/0.log" Jul 16 21:20:35.816222 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.816194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/cni-plugins/0.log" Jul 16 21:20:35.839581 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.839550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/bond-cni-plugin/0.log" Jul 16 21:20:35.871072 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.871048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/routeoverride-cni/0.log" Jul 16 21:20:35.896574 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.896544 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/whereabouts-cni-bincopy/0.log" Jul 16 21:20:35.920885 ip-10-0-131-179 kubenswrapper[2564]: I0716 21:20:35.920865 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s2gq9_c7d430ca-5628-4814-9959-e63438410d86/whereabouts-cni/0.log"